An optimized method for chip package quality detection based on 2D and 3D composite imaging

Abstract

The present invention discloses an optimization method for chip package quality detection based on 2D and 3D composite imaging, comprising the following steps: S1, collecting two-dimensional images and three-dimensional structure images of the chip package area; S2, performing spatial registration and normalization processing, extracting brightness gradients, depth gradients and structural curvatures, and generating an image fusion feature set; S3, constructing an improved Mumford-Shah image energy functional model based on the image fusion feature set; S4, setting up a spatial attention adjustment mechanism to adjust the weight of the boundary length term; S5, performing variational optimization to generate an edge response map, and extracting the boundary contour to form a chip package structure boundary map; S6, fusing the boundary map and the three-dimensional structure image to construct a boundary map spectrum, and extracting the graph structure features; S7, constructing a package quality scoring function, and outputting the defect type, defect location and package quality scoring value. The present invention fuses 2D and 3D structure features, and improves the modeling ability of chip package defect recognition and scoring.

Description

Technical Field

[0001] The present invention relates to the technical field of semiconductor manufacturing inspection, and particularly to an optimization method for chip package quality inspection based on 2D and 3D composite imaging. Background Art

[0002] In the field of semiconductor manufacturing, chip package quality is an important indicator affecting the reliability and service life of integrated circuits. Especially in the context of the widespread application of advanced packaging forms such as high-density packaging and three-dimensional stacked packaging, higher requirements are put forward for the detection accuracy and coverage rate of chip package defects. Currently, the commonly used package quality inspection technologies mainly rely on two-dimensional visible light images or X-ray images, and identify geometric anomalies or connection defects of package structures such as solder joints, pins, and bonding wires through image processing algorithms. Common image processing methods include edge detection, morphological analysis, template matching, and convolutional neural network classification, etc., which have certain recognition capabilities in scenarios with good lighting conditions and clear structures. However, for non-surface visible defects such as multi-layer structure superposition, slight deformation caused by internal stress, pad occlusion, and false soldering, the detection ability is significantly insufficient.

[0003] In order to improve the ability to obtain deep structure information, some research has introduced three-dimensional structure imaging means, such as structured light scanning, point cloud reconstruction, and laser ranging, etc., to obtain the three-dimensional depth image of the chip package structure, and then combine it with the two-dimensional image to form a detection mode of multi-source information fusion. This 2D and 3D composite detection idea can, to a certain extent, make up for the problem of insufficient defect expression from a single perspective. In the prior art, depth maps and two-dimensional images are often registered at the pixel level, and then fused features are generated through simple image superposition, average filtering, or geometric feature stitching, and then input into traditional classification networks or threshold-based rule algorithms for defect classification. However, in cases where the package structure is complex, the boundaries are irregular, the surface reflection or occlusion interference is serious, such fusion methods cannot effectively extract spatial boundary information, and lack the ability to depict the continuity and spatial consistency of package defects, resulting in the risk of false detection and missed detection in the final defect determination.

[0004] In addition, in the existing image modeling mechanism, edge detection operators or graph cut algorithms based on gray-scale changes are generally used as tools for extracting package structure boundaries. Such methods are difficult to adapt to the problems of gray-scale non-uniformity, structural density change, and noise disturbance in package images. The classic Mumford-Shah energy model provides a mathematical optimization framework for image segmentation, which can consider image smoothing, boundary accuracy, and image reconstruction consistency at the same time. However, it is limited in scenarios with serious gray-scale drift or complex structures. The traditional model fails to adjust the boundary term response by combining three-dimensional structure information, so its applicability in the chip package inspection task is insufficient.

[0005] On the other hand, although some methods introduce deep neural networks for defect classification, their training process is highly sensitive to the quantity of the dataset and the annotation accuracy, and the model interpretability is poor, making it difficult to meet the requirements of industrial package detection scenarios that strongly rely on traceability and physical rationality. Existing models generally lack the ability to model the coherence and consistency of the package structure boundary in three-dimensional space. Especially when there are structural defects such as solder joint defects, metal layer offsets, and bonding misalignments, they cannot expand local anomalies into global judgment bases through graph structures or spatial constraints, resulting in fragmented defect boundary determination and ambiguous scoring mechanisms.

[0006] Therefore, how to provide an optimized method for chip package quality detection based on 2D and 3D composite imaging is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0007] An object of the present invention is to propose an optimized method for chip package quality detection based on 2D and 3D composite imaging. The present invention fully integrates spatial registration processing of two-dimensional images and three-dimensional structure images, image energy functional modeling, boundary atlas construction, and graph structure feature analysis methods, and details the process of extracting the package structure boundary through an improved Mumford-Shah model and identifying defect types and scoring package quality based on the graph structure, with the advantages of clear feature expression, accurate boundary modeling, and strong robustness of the scoring mechanism.

[0008] An optimized method for chip package quality detection based on 2D and 3D composite imaging according to an embodiment of the present invention includes the following steps:

[0009] S1. Collect two-dimensional images and three-dimensional structure images of the chip package area;

[0010] S2. Perform spatial registration and normalization processing on the two-dimensional images and three-dimensional structure images, extract luminance gradients, depth gradients, and structural curvatures, and generate an image fusion feature set;

[0011] S3. Based on the image fusion feature set, construct an image energy functional model, and the image energy functional model is an improved Mumford-Shah model, including a region smoothing term, a reconstruction fidelity term, and a boundary length term;

[0012] S4. Set a spatial attention adjustment mechanism in the image energy functional model to spatially adjust the weight of the boundary length term according to different structure distributions in the chip package area;

[0013] S5. Use a variational optimization method to solve the image energy functional model, generate an edge response map, extract the boundary contour of the chip package area, and form a chip package structure boundary map;

[0014] S6. Merge the boundary map of the chip packaging structure with the 3D structure image to construct a boundary map of the chip packaging area, and extract boundary continuity features, boundary consistency features, and structural anomaly features;

[0015] S7. Construct a packaging quality scoring function, using the boundary continuity features, boundary consistency features, and structural anomaly features as inputs to generate the defect type, defect location, and packaging quality score value of the chip packaging area.

[0016] Optionally, the two-dimensional image includes a visible light image or an X-ray image, and the 3D structure image includes a depth image obtained by a point cloud scanning device or a depth sensor.

[0017] Optionally, the regional smoothing term is constructed based on the luminance gradient and depth gradient in the image fusion feature set, and is defined within the chip packaging area outside the boundary map.

[0018] Optionally, the reconstruction fidelity term is constructed based on the difference between the pixel values of the two-dimensional image and the output values of the image model, and is defined throughout the chip packaging area.

[0019] Optionally, the specific steps of S2 include:

[0020] S21. Perform spatial registration on the two-dimensional image and the 3D structure image to establish the correspondence between the two-dimensional image coordinate system and the 3D structure image coordinate system;

[0021] S22. Normalize the spatially registered two-dimensional image and 3D structure image respectively, and standardize the pixel values in the two-dimensional image and the depth values in the 3D structure image to the interval [0, 1];

[0022] S23. Extract the luminance gradient from the normalized two-dimensional image, and extract the depth gradient and structural curvature from the normalized 3D structure image;

[0023] S24. At the coordinate position (x, y), construct an image fusion feature set vector , where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the luminance gradient, is the depth gradient, is the structural curvature.

[0024] Optionally, the specific steps of S3 include:

[0025] S31. Set the image gray function:

[0026] ;

[0027] Wherein, x is the horizontal coordinate of the image, y is the vertical coordinate of the image, and I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y). is a two-dimensional Gaussian kernel function with a standard deviation of ; is the standard deviation of the Gaussian kernel, is the convolution operator, and u(x, y) is the image grayscale function;

[0028] S32. Set the boundary weighting function:

[0029] ;

[0030] Wherein, is the boundary weighting function, is the depth gradient of the three-dimensional structure image at the coordinate (x, y), is the Euclidean norm of the depth gradient, is the structure curvature, is the absolute value of the structure curvature, is the boundary reference weight coefficient, is the adjustment coefficient of the depth guidance factor, is the adjustment coefficient of the structure curvature adjustment function;

[0031] S33. Construct an image energy functional model:

[0032] ;

[0033] Wherein, is the image energy functional model, is the two-dimensional domain of the chip packaging area, is the set of boundary curves, is the square of the gradient modulus of the image grayscale function at the coordinate (x, y), and ds is the infinitesimal length on the integral path along the boundary curve, , , are the non-negative weighting coefficients of the regional smoothing term, the reconstruction fidelity term, and the boundary length term, respectively;

[0034] S34. The boundary length term is constructed by performing a path integral on the boundary curve set on the boundary weighting function . The boundary weighting function contains a depth guidance factor and a structure curvature adjustment function calculated from the three-dimensional structure image;

[0035] S35. The image energy functional model takes the image fusion feature set as the input, the image grayscale function as the variational optimization object, the boundary weighting function as the integrand of the boundary integral term, and the image energy functional model serves as the basic structure for generating the edge response map and constructing the chip packaging structure boundary map.

[0036] Optionally, S4 specifically includes:

[0037] S41. Set the spatial adjustment function:

[0038] ;

[0039] where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the spatial adjustment reference coefficient, , are the spatial adjustment factors, is the boundary density function of the chip packaging structure at the coordinate (x, y), is the structure complexity function of the chip packaging structure at the coordinate (x, y);

[0040] S42. Based on the spatial adjustment function, adjust the boundary weighting function in the image energy functional model, and set the boundary weighting function after spatial adjustment:

[0041] ;

[0042] where, is the boundary weighting function, is the boundary weighting function after spatial adjustment, is the spatial adjustment function;

[0043] S43. Replace the integrand in the boundary length term with the boundary weighting function after spatial adjustment, and update the boundary length term expression to , where, is the set of boundary curves extracted from the chip packaging area, and ds is the path differential length on the boundary curve.

[0044] Optionally, S5 specifically includes:

[0045] S51. Set the image energy functional model as the objective function of variational optimization:

[0046] ;

[0047] where, is the image energy functional model, u(x, y) is the image gray level function, I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y), is the square of the gradient modulus of the image gray level function, is the boundary weighting function after spatial adjustment, is the set of boundary curves, is the two-dimensional domain of the chip packaging area, , , is a non - negative weighting coefficient, ds is the differential length of the integration path along the boundary curve, x is the horizontal coordinate of the image, and y is the vertical coordinate of the image;

[0048] S52. Perform variational optimization on the image energy functional model to obtain an edge response map. The edge response map is a two - dimensional matrix with the same size as the original image, and each pixel position corresponds to a boundary response value;

[0049] S53. Extract the boundary contour of the chip package area according to the comparison result between the pixel value in the edge response map and the boundary response threshold. The boundary contour is composed of a set of pixel coordinate points that meet the response conditions;

[0050] S54. Represent the pixel coordinate points in the boundary contour as a coordinate sequence , where is the abscissa of the i - th boundary point, is the ordinate of the i - th boundary point, and i is a positive integer index value;

[0051] S55. Arrange the coordinate sequence according to the boundary connectivity relationship to construct a boundary map of the chip package structure. The boundary map of the chip package structure is represented by a set of boundary curves .

[0052] Optionally, the specific steps of S6 are as follows:

[0053] S61. Extract coordinates from the boundary map of the chip package structure to obtain a set of boundary coordinates , where, is the horizontal coordinate of the i - th boundary point, is the vertical coordinate of the i - th boundary point, N is the number of boundary points, and i is a positive integer index;

[0054] S62. Map the set of boundary coordinates to the three - dimensional structure image, and obtain the depth value at each boundary point to form a three - dimensional boundary point set , where, is the output value of the depth function of the three - dimensional structure image at the coordinate , and is the depth value of the corresponding point;

[0055] S63. Based on the three - dimensional boundary point set P, construct a boundary graph of the chip package area. Set the boundary graph structure as G=(V, E), where V is the set of nodes and E is the set of edges. The node represents the three - dimensional boundary point , and the edge represents the connection relationship between adjacent nodes;

[0056] S64. Calculate the boundary continuity feature on graph G:

[0057] ;

[0058] where, is the boundary continuity feature, representing the average value of the square of the spatial Euclidean distance between adjacent three-dimensional boundary points, represents the node and is the modulus of the three-dimensional coordinate difference, and N is the number of boundary points;

[0059] S65. Calculate the boundary consistency feature on graph G:

[0060] ;

[0061] where, is the boundary consistency feature, representing the entropy value of the boundary normal vector direction distribution, K is the number of direction discretization intervals, is the normalized frequency of the boundary normal vector in the j-th direction interval, and the lower the entropy value, the more concentrated the direction;

[0062] S66. Calculate the structural anomaly feature on graph G:

[0063] ;

[0064] where, is the structural anomaly feature, represents the modulus of the depth gradient of the three-dimensional structure image at point , is the structural curvature calculated at point .

[0065] Optionally, the S7 specifically includes:

[0066] S71. Construct an encapsulation quality scoring function:

[0067] ;

[0068] where, Q is the encapsulation quality score value of the chip encapsulation area, is the boundary continuity feature, is the boundary consistency feature, is the structural anomaly feature, , , are non-negative weighting coefficients of the encapsulation quality scoring function;

[0069] S72. Determine the defect type label for each boundary point based on the boundary point coordinates in the boundary map of the chip package structure, the corresponding depth values in the three-dimensional structure image, and the graph structure feature vector, to obtain the defect type label;

[0070] S73. Associate the spatial coordinates, defect type label of each boundary point with the package quality score value to generate a result set, and the result set includes the defect positions, defect types, and package quality score values of each boundary point in the chip package area.

[0071] The beneficial effects of the present invention are as follows:

[0072] By constructing an optimized method for chip package quality detection based on 2D and 3D composite imaging, the present invention realizes accurate boundary modeling and structural feature extraction of the package area on the basis of spatial registration and fusion of two-dimensional images and three-dimensional structure images. Compared with the existing detection methods that rely on a single image dimension or low-level feature stitching, the present invention adopts an image fusion feature set that combines luminance gradient, depth gradient, and structural curvature, effectively enhancing the identification ability between the package boundary and the internal structure. At the same time, an improved Mumford-Shah image energy functional model is introduced, which combines the boundary weighting function with the spatial attention adjustment mechanism to construct a variational optimization framework including a regional smoothing term, a reconstruction fidelity term, and a boundary length term, improving the boundary response accuracy for gray-scale inhomogeneous regions and positions with sudden changes in structural curvature.

[0073] In addition, by fusing the edge response map with the three-dimensional structure image, the present invention constructs a boundary map of the chip package area, and further extracts boundary continuity, boundary consistency, and structural anomaly features to construct a graph structure feature vector, enabling the package quality assessment to take into account the joint expression of structural integrity and local anomalies. The package quality scoring function is based on a weighted linear combination form to form an adjustable parameter model, which can quantitatively evaluate the influence range of different types of structural defects on the score value. The identification of defect types is realized based on the linkage between the boundary map and the graph structure features, improving the identification ability for minor defects and spatial asymmetric structural anomalies. Description of the Drawings

[0074] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0075] Figure 1 is the overall flowchart of an optimized method for chip package quality detection based on 2D and 3D composite imaging proposed by the present invention;

[0076] Figure 2Schematic diagram for constructing an image energy functional model and extracting boundaries of an optimized method for chip package quality inspection based on 2D and 3D composite imaging proposed by the present invention;

[0077] Figure 3 Schematic diagram for constructing a boundary map and outputting a package quality score of an optimized method for chip package quality inspection based on 2D and 3D composite imaging proposed by the present invention. Detailed implementation manners

[0078] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0079] Refer to Figures 1-3 , an optimized method for chip package quality inspection based on 2D and 3D composite imaging, comprising the following steps:

[0080] S1. Collect a two-dimensional image and a three-dimensional structure image of the chip package area;

[0081] S2. Perform spatial registration and normalization processing on the two-dimensional image and the three-dimensional structure image, extract the luminance gradient, depth gradient, and structural curvature, and generate an image fusion feature set;

[0082] S3. Construct an image energy functional model based on the image fusion feature set. The image energy functional model is an improved Mumford-Shah model, including a regional smoothing term, a reconstruction fidelity term, and a boundary length term;

[0083] S4. Set a spatial attention adjustment mechanism in the image energy functional model, and adjust the weight of the boundary length term spatially according to the different structure distributions in the chip package area;

[0084] S5. Use a variational optimization method to solve the image energy functional model, generate an edge response map, extract the boundary contour of the chip package area, and form a chip package structure boundary map;

[0085] S6. Fusion the chip package structure boundary map with the three-dimensional structure image, construct a boundary map of the chip package area, and extract boundary continuity features, boundary consistency features, and structural anomaly features;

[0086] S7. Construct a package quality scoring function, use the boundary continuity features, boundary consistency features, and structural anomaly features as inputs, and generate the defect type, defect location, and package quality score value of the chip package area.

[0087] The present invention constructs a two-dimensional image and three-dimensional structure image acquisition process for the chip packaging area, establishing a multi-modal information basis, providing complete input conditions for subsequent image fusion and structural boundary analysis, and enhancing the model's ability to obtain packaging information.

[0088] In this embodiment, the two-dimensional image includes a visible light image or an X-ray image, and the three-dimensional structure image includes a depth image obtained by a point cloud scanning device or a depth sensor.

[0089] The present invention clearly defines the source types of the two-dimensional image and the three-dimensional structure image, and adopts spatial registration and normalization processing means to ensure the extraction of luminance gradient, depth gradient and structural curvature in a unified coordinate system, improving the accuracy of feature expression and the stability of the fusion effect.

[0090] In this embodiment, the regional smoothness term is constructed based on the luminance gradient and depth gradient in the image fusion feature set, and is defined within the chip packaging area outside the boundary map.

[0091] The present invention constructs a regional smoothness term based on the luminance gradient and depth gradient, and defines its scope of action within the packaging area outside the boundary map, enabling the energy model to perform continuous modeling on the internal feature changes in the packaging area and enhancing the processing ability for the structural transition area.

[0092] In this embodiment, the reconstruction fidelity term is constructed based on the difference between the pixel values of the two-dimensional image and the output values of the image model, and is defined over the entire chip packaging area.

[0093] The present invention constructs a reconstruction fidelity term using the difference between the pixel values of the two-dimensional image and the model output values, and defines this term over the entire chip packaging area, effectively maintaining the fidelity ability of the image model to the original image structure information and improving the overall stability of the model.

[0094] In this embodiment, S2 specifically includes:

[0095] S21. Perform spatial registration on the two-dimensional image and the three-dimensional structure image to establish the correspondence between the two-dimensional image coordinate system and the three-dimensional structure image coordinate system;

[0096] S22. Respectively perform normalization processing on the spatially registered two-dimensional image and three-dimensional structure image, and standardize the pixel values in the two-dimensional image and the depth values in the three-dimensional structure image to the interval [0, 1];

[0097] S23. Extract the luminance gradient in the normalized two-dimensional image, and extract the depth gradient and structural curvature in the normalized three-dimensional structure image;

[0098] S24. Construct an image fusion feature set vector at the coordinate position (x, y). where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the luminance gradient, is the depth gradient, is the structure curvature.

[0099] The present invention details the construction process of the image fusion feature set, including the spatial registration of two-dimensional and three-dimensional images, normalization processing, luminance and depth gradient calculation, and structure curvature extraction, establishing a clear and executable preprocessing process for easy application in the encapsulation detection task.

[0100] In this embodiment, the S3 specifically includes:

[0101] S31. Set the image grayscale function:

[0102] ;

[0103] where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y), is the two-dimensional Gaussian kernel function with a standard deviation of , is the standard deviation of the Gaussian kernel, is the convolution operator, and u(x, y) is the image grayscale function;

[0104] S32. Set the boundary weighting function:

[0105] ;

[0106] where is the boundary weighting function, is the depth gradient of the three-dimensional structure image at the coordinate (x, y), is the Euclidean norm of the depth gradient, is the structure curvature, is the absolute value of the structure curvature, is the boundary reference weight coefficient, is the adjustment coefficient of the depth guidance factor, is the adjustment coefficient of the structure curvature adjustment function;

[0107] S33. Construct the image energy functional model:

[0108] ;

[0109] where is the image energy functional model, is the two-dimensional domain of the chip packaging area, is a set of boundary curves, is the square of the gradient modulus of the image grayscale function at the coordinates (x, y), and ds is the infinitesimal length on the integral path along the boundary curve. , , are non - negative weighting coefficients of the regional smoothing term, the reconstruction fidelity term, and the boundary length term, respectively;

[0110] S34. The boundary length term is constructed by performing a path integral on the boundary curve set for the boundary weighting function . The boundary weighting function contains a depth guidance factor and a structure curvature adjustment function calculated from the three - dimensional structure image;

[0111] S35. The image energy functional model takes the image fusion feature set as the input, the image grayscale function as the variational optimization object, the boundary weighting function as the integrand of the boundary integral term, and the image energy functional model serves as the basic structure for generating the edge response map and constructing the boundary map of the chip packaging structure.

[0112] In the present invention, by setting the image grayscale function and the boundary weighting function, and introducing the structure curvature adjustment function and the depth guidance factor, an improved Mumford - Shah energy functional model is constructed, enhancing the model's response ability to three - dimensional boundary changes and forming the basis for subsequent optimization and solution.

[0113] In this embodiment, the specific content of S4 includes:

[0114] S41. Set the spatial adjustment function:

[0115] ;

[0116] where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the spatial adjustment reference coefficient, , are the spatial adjustment factors, is the boundary density function of the chip packaging structure boundary at the coordinates (x, y), is the structure complexity function of the chip packaging structure at the coordinates (x, y);

[0117] S42. Adjust the boundary weighting function in the image energy functional model based on the spatial adjustment function, and set the spatially adjusted boundary weighting function:

[0118] ;

[0119] where, is the boundary weighting function, is the boundary weighting function after spatial adjustment, is a spatial adjustment function;

[0120] S43. Replace the integrand in the boundary length term with the spatially adjusted boundary weighting function, and update the boundary length term expression to , where is the set of boundary curves extracted from the chip packaging area, and ds is the path differential length on the boundary curve.

[0121] The present invention sets a spatial attention adjustment function and regulates the boundary weighting function, enabling the boundary length term to have an adaptive adjustment ability in different packaging structure distribution areas, thereby improving the expression accuracy of the boundary curve by the model in complex structure areas.

[0122] In this embodiment, the S5 specifically includes:

[0123] S51. Set the image energy functional model as the objective function for variational optimization:

[0124] ;

[0125] where is the image energy functional model, u(x, y) is the image gray level function, I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y), is the square of the gradient modulus of the image gray level function, is the spatially adjusted boundary weighting function, is the set of boundary curves, is the two-dimensional domain of the chip packaging area, , , are non-negative weighting coefficients, ds is the differential length of the integration path along the boundary curve, x is the horizontal coordinate of the image, and y is the vertical coordinate of the image;

[0126] S52. Perform variational optimization on the image energy functional model to obtain an edge response map, which is a two-dimensional matrix with the same size as the original image, and each pixel position corresponds to a boundary response value;

[0127] S53. Extract the boundary contour of the chip packaging area according to the comparison result between the pixel value in the edge response map and the boundary response threshold, and the boundary contour is composed of a set of pixel coordinate points that meet the response conditions;

[0128] S54. Represent the pixel coordinate points in the boundary contour as a coordinate sequence , where is the abscissa of the i-th boundary point, is the ordinate of the i-th boundary point, and i is a positive integer index value;

[0129] S55. Arrange the coordinate sequence According to the boundary connectivity relationship to construct the boundary graph of the chip package structure, and the boundary graph of the chip package structure is represented by a set of boundary curves .

[0130] The present invention solves the image energy functional model by a variational optimization method, generates an edge response map, extracts the boundary contour of the package structure in combination with the response value, and further constructs the boundary graph of the chip package structure, realizing the structural conversion process from the fusion feature to the boundary graph.

[0131] In this embodiment, the specific steps of S6 are as follows:

[0132] S61. Extract the coordinates of the boundary graph of the chip package structure to obtain a set of boundary coordinates , where is the horizontal coordinate of the i-th boundary point, is the vertical coordinate of the i-th boundary point, N is the number of boundary points, and i is a positive integer index;

[0133] S62. Map the set of boundary coordinates to the three-dimensional structure image, and obtain the depth value at each boundary point to form a three-dimensional boundary point set , where is the output value of the depth function of the three-dimensional structure image at the coordinate , and is the depth value of the corresponding point;

[0134] S63. Based on the three-dimensional boundary point set P, construct the boundary map of the chip package area, and set the boundary map structure as G=(V, E), where V is the set of nodes and E is the set of edges. The node represents the three-dimensional boundary point , and the edge represents the connection relationship between adjacent nodes;

[0135] S64. Calculate the boundary continuity feature on the graph G:

[0136] ;

[0137] where is the boundary continuity feature, representing the average value of the square of the spatial Euclidean distance between adjacent three-dimensional boundary points, represents the modulus of the three-dimensional coordinate difference between the node and , and N is the number of boundary points;

[0138] S65. Calculate the boundary consistency feature on the graph G:

[0139] ;

[0140] Among them, is the boundary consistency feature, representing the entropy value of the boundary normal vector direction distribution, K is the number of direction discretization intervals, is the normalized frequency of the boundary normal vector in the j-th direction interval, and the lower the entropy value, the more concentrated the direction;

[0141] S66. Calculate the structural anomaly feature on the graph G:

[0142] ;

[0143] Among them, is the structural anomaly feature, represents the magnitude of the depth gradient at the point in the three-dimensional structure image, is the point The structural curvature calculated at the location.

[0144] The present invention is based on the joint modeling of the three-dimensional structure image and the chip package structure boundary map, establishes a three-dimensional boundary point set and a spectral structure, and extracts boundary continuity, boundary consistency and structural anomaly features in the graph to form a quantifiable graph structure feature vector, providing support for defect determination.

[0145] In this embodiment, the specific steps of S7 are as follows:

[0146] S71. Construct a package quality scoring function:

[0147] ;

[0148] Among them, Q is the package quality score value of the chip package area, is the boundary continuity feature, is the boundary consistency feature, is the structural anomaly feature, , , are non-negative weighting coefficients of the package quality scoring function;

[0149] S72. Based on the boundary point coordinates in the chip package structure boundary map, the corresponding depth values in the three-dimensional structure image, and the graph structure feature vector, determine the defect type for each boundary point to obtain a defect type label;

[0150] S73. Associate the spatial coordinates, defect type label and package quality score value of each boundary point to generate a result set, and the result set includes the defect location, defect type and package quality score value of each boundary point in the chip package area.

[0151] The present invention constructs a packaging quality scoring function, which takes three types of graph structure features as input to generate a scoring value, and based on the scoring result and structural information, labels the defect type and location, realizing the linkage expression of quantitative evaluation of chip packaging quality and spatial defect localization.

[0152] Example 1:

[0153] To verify the feasibility of the present invention in implementation, the present invention is applied to the chip packaging detection task of a certain chip manufacturing enterprise in the fourth quarter of 2024. The packaging production line of this enterprise adopts a hybrid packaging process of high-density flip-chip bonding and multi-layer BGA, and the products are widely used in the fields of consumer electronics and industrial control. Due to the complex packaging structure and the existence of multi-layer stacking, the traditional detection method based only on two-dimensional images has poor effect in identifying structural defects such as micro-warpage, pad misalignment and solder joint voids, with a high missed detection rate, seriously affecting the chip yield.

[0154] In this packaging detection scenario, the detection system is deployed at the detection station of Production Line 1 in the factory area, and mainly consists of a dual-mode imaging module, an image processing and analysis module, and a scoring feedback module. The dual-mode imaging module simultaneously collects visible light images and structured light scanning depth maps of the chip packaging area, with an image size of 2048×2048 pixels and a depth accuracy of 0.05 mm. Image registration adopts the method based on SIFT feature point matching and RANSAC correction, and the normalization range is unified to [0, 1]. Subsequently, brightness gradient, depth gradient and structural curvature are extracted to construct an image fusion feature set.

[0155] In the image analysis stage, the system calls an image energy functional model based on the improved Mumford-Shah model, which includes a region smoothing term, a reconstruction fidelity term and a boundary length term. The boundary length term introduces a regulation function based on structural curvature and a guiding factor based on depth gradient, enhancing the model's response ability to depth perturbation and packaging boundary mutation. At the same time, to adapt to the influence of different packaging structure distributions on boundary expression, the system introduces a spatial attention regulation mechanism to adjust the boundary weighting function regionally. The model optimization adopts a variational solution algorithm to iterate and converge, and finally outputs an edge response map and a chip packaging structure boundary map.

[0156] The system fuses the boundary map and the depth map to construct a boundary map spectrum of the chip packaging area, extracts boundary continuity, boundary consistency and structural anomaly features based on the graph structure, generates a three-dimensional structure feature vector. The scoring function outputs the packaging quality scoring value in the way of three-item weighted combination, with the score range of [0, 100], and identifies the defect type and location according to the structural change and local abnormal points.

[0157] In mid-December 2024, the system detected a total of 800 samples on the production line of encapsulated chip products with a batch number of FC-MD422, among which 200 were defective samples marked by manual review. The total time consumed by the method of the present invention for detection is 4.3 seconds per piece, and the corresponding average time for manual detection is 9.1 seconds per piece. In terms of defect recognition accuracy, the comparison with the manual marking results is shown in Table 1:

[0158] Table 1 Defect Detection Statistical Table

[0159] ;

[0160] In addition, in terms of the graph structure scoring, the comparison between the encapsulation scoring results and the manual quality inspection scoring shows that the average deviation between the scoring value of this method and the manual scoring result is 1.74 points (out of 100 full marks), and the scoring consistency is better than that of the traditional deep learning detection model (the deviation is 4.91 points). The scoring output examples are shown in Table 2:

[0161] Table 2 Chip Scoring and Defect Recognition Example Table

[0162] ;

[0163] It can be seen from the data that the method of the present invention demonstrates excellent structure recognition ability, boundary modeling accuracy, and scoring stability in the actual production scenario. It not only significantly outperforms manual detection and traditional deep models in terms of detection time, but also achieves a substantial improvement in defect recognition accuracy and scoring consistency.

[0164] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes, shall be covered by the protection scope of the present invention.

Claims

1. An optimized method for detecting the quality of chip packaging based on 2D and 3D composite imaging, characterized in that, It includes the following steps: S1. Collect the two-dimensional image and three-dimensional structure image of the chip packaging area; S2. Perform spatial registration and normalization processing on the two-dimensional image and three-dimensional structure image, extract the brightness gradient, depth gradient and structural curvature, and generate an image fusion feature set; S3. Based on the image fusion feature set, construct an image energy functional model, and the image energy functional model is an improved Mumford-Shah model, which includes a regional smoothing term, a reconstruction fidelity term and a boundary length term; S4. Set a spatial attention adjustment mechanism in the image energy functional model, and perform spatial adjustment on the weight of the boundary length term according to the different structure distributions in the chip packaging area; S5. Use the variational optimization method to solve the image energy functional model, generate an edge response map, extract the boundary contour of the chip packaging area, and form a chip packaging structure boundary map; S6. Fuse the chip packaging structure boundary map with the three-dimensional structure image, construct a boundary map of the chip packaging area, and extract the boundary continuity feature, boundary consistency feature and structural anomaly feature; S7. Construct a packaging quality scoring function, use the boundary continuity feature, boundary consistency feature and structural anomaly feature as inputs, and generate the defect type, defect location and packaging quality score value of the chip packaging area.

2. The optimized method for chip package quality detection based on 2D and 3D composite imaging according to claim 1, wherein The two-dimensional image includes a visible light image or an X-ray image, and the three-dimensional structure image includes a depth image obtained by a point cloud scanning device or a depth sensor.

3. An optimized method for detecting the quality of chip packaging based on 2D and 3D composite imaging according to claim 1, characterized in that The regional smoothing term is constructed based on the brightness gradient and depth gradient in the image fusion feature set, and is defined in the chip packaging area outside the boundary map.

4. An optimized method for detecting the quality of chip packaging based on 2D and 3D composite imaging according to claim 1, characterized in that, The reconstruction fidelity term is constructed based on the difference between the pixel value of the two-dimensional image and the output value of the image model, and is defined in the entire chip packaging area.

5. An optimized method for detecting the quality of chip packaging based on 2D and 3D composite imaging according to claim 1, characterized in that, The specific content of S2 includes: S21. Perform spatial registration on the two-dimensional image and three-dimensional structure image, and establish the corresponding relationship between the two-dimensional image coordinate system and the three-dimensional structure image coordinate system; S22. Respectively perform normalization processing on the spatially registered two-dimensional image and three-dimensional structure image, and standardize the pixel value in the two-dimensional image and the depth value in the three-dimensional structure image to the interval [0,1]; S23. Extract the brightness gradient in the normalized two-dimensional image, and extract the depth gradient and structural curvature in the normalized three-dimensional structure image; S24. Construct an image fusion feature set vector at the coordinate position (x, y), where x is the horizontal coordinate of the image and y is the vertical coordinate of the image, is the brightness gradient, is the depth gradient, is the structural curvature.

6. The optimized method for detecting the quality of chip packaging based on 2D and 3D composite imaging according to claim 1, characterized in that, The specific content of S3 includes: S31. Set the image gray function: ; where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, and I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y). is a two-dimensional Gaussian kernel function with a standard deviation of , is the standard deviation of the Gaussian kernel, is the convolution operator, and u(x, y) is the image grayscale function; S32. Set the boundary weighting function: ; Among them, is the boundary weighting function, is the depth gradient of the three-dimensional structure image at the coordinates (x, y), is the Euclidean norm of the depth gradient, is the structure curvature, is the absolute value of the structure curvature, is the boundary reference weight coefficient, is the adjustment coefficient of the depth guiding factor, is the adjustment coefficient of the structure curvature adjustment function; S33. Construct the image energy functional model: ; Among them, is the image energy functional model, is the two-dimensional domain of the chip packaging area, is the set of boundary curves, is the square of the gradient modulus of the image grayscale function at the coordinates (x, y), and ds is the differential element length on the integral path along the boundary curve, and and are the non-negative weighting coefficients of the regional smoothing term, the reconstruction fidelity term, and the boundary length term, respectively; S34. The boundary length term is constructed by performing a path integral on the boundary curve set for the boundary weighting function . The boundary weighting function includes a depth guidance factor and a structure curvature adjustment function calculated from the three-dimensional structure image; S35. The image energy functional model takes the image fusion feature set as the input, the image gray function as the variational optimization object, the boundary weighting function as the integrand of the boundary integral term, and the image energy functional model as the basic structure for generating the edge response map and constructing the chip packaging structure boundary map.

7. An optimized method for chip package quality inspection based on 2D and 3D composite imaging according to claim 1, characterized in that The specific content of S4 includes: S41. Set the spatial adjustment function: ; where x is the horizontal coordinate of the image and y is the vertical coordinate of the image, is the spatial adjustment reference coefficient, and are the spatial adjustment factors, is the boundary density function of the chip package structure at the coordinate (x, y), is the structure complexity function of the chip package structure at the coordinate (x, y); S42. Adjust the boundary weighting function in the image energy functional model based on the spatial adjustment function, and set the spatially adjusted boundary weighting function: ; Among them, is the boundary weighting function, is the boundary weighting function after spatial adjustment, is the spatial adjustment function; S43. Replace the integrand in the boundary length term with the spatially adjusted boundary weighting function, and update the boundary length term expression to , where is the set of boundary curves extracted from the chip packaging area, and ds is the path differential length on the boundary curve.

8. An optimized method for chip package quality detection based on 2D and 3D composite imaging according to claim 1, characterized in that, The specific content of S5 includes: S51. Set the image energy functional model as the objective function of variational optimization; S52. Perform variational optimization on the image energy functional model to obtain an edge response map, which is a two-dimensional matrix with the same size as the original image, and each pixel position corresponds to a boundary response value; S53. Extract the boundary contour of the chip packaging area according to the comparison result between the pixel value in the edge response map and the boundary response threshold. The boundary contour is composed of a set of pixel coordinate points that meet the response conditions; S54. Represent the pixel coordinate points in the boundary contour as a coordinate sequence , where is the abscissa of the i-th boundary point, is the ordinate of the i-th boundary point, and i is a positive integer index value; S55. Arrange the coordinate sequence in accordance with the boundary connection relationship to construct a boundary graph of the chip package structure, where the boundary graph of the chip package structure is represented by a set of boundary curves .

9. The optimized method for chip package quality detection based on 2D and 3D composite imaging according to claim 1, characterized in that, The specific content of S6 includes: S61. Extract coordinates from the boundary map of the chip packaging structure to obtain a set of boundary coordinates , where is the horizontal coordinate of the i-th boundary point, is the vertical coordinate of the i-th boundary point, N is the number of boundary points, and i is a positive integer index; S62. Map the set of boundary coordinates to the three-dimensional structure image, and obtain the depth value at each boundary point to form a three-dimensional boundary point set . Among them, is the output value of the depth function of the three-dimensional structure image at the coordinate , and is the depth value of the corresponding point;​ S63. Construct a boundary map of the chip packaging area based on the three-dimensional boundary point set P, and set the boundary map structure as graph G=(V,E), where V is the set of nodes and E is the set of edges. The nodes represent three-dimensional boundary points , and the edges represent the connection relationships between adjacent nodes; S64. Calculate the boundary continuity feature on graph G: ; Among them, is the boundary continuity feature, representing the average value of the squared spatial Euclidean distance between adjacent three-dimensional boundary points, represents the node and is the modulus of the three-dimensional coordinate difference of, and N is the number of boundary points; S65. Calculate the boundary consistency feature on graph G: ; Among them, is the boundary consistency feature, representing the entropy value of the boundary normal vector direction distribution, and K is the number of direction discretization intervals. is the normalized frequency of the boundary normal vector in the j-th direction interval. The lower the entropy value, the more concentrated the direction is. S66. Calculate the structural anomaly feature on graph G: ; Among them, is the structural anomaly feature, represents the magnitude of the depth gradient of the three-dimensional structure image at the point and is the structural curvature calculated at the point .

10. An optimized method for chip package quality detection based on 2D and 3D composite imaging according to claim 1, characterized in that, The specific content of S7 includes: S71. Construct a packaging quality scoring function: ; Among them, Q is the encapsulation quality score value of the chip encapsulation area, is the boundary continuity feature, is the boundary consistency feature, is the structural anomaly feature, , , are non - negative weighting coefficients of the encapsulation quality scoring function; S72. Based on the boundary point coordinates in the chip packaging structure boundary map, the corresponding depth values in the three-dimensional structure image, and the graph structure feature vector, determine the defect type for each boundary point to obtain a defect type label; S73. Associate the spatial coordinates, defect type labels, and packaging quality scoring values of each boundary point to generate a result set, which contains the defect positions, defect types, and packaging quality scoring values of each boundary point in the chip packaging area.

Images