Chip packaging quality detection optimization method based on 2D and 3D composite imaging

Abstract

The invention discloses a chip packaging quality detection optimization method based on 2D and 3D composite imaging. The method comprises the following steps: S1, collecting a two-dimensional image and a three-dimensional structure image of a chip packaging area; s2, carrying out spatial registration and normalization processing, extracting a brightness gradient, a depth gradient and a structure curvature, and generating an image fusion feature set; s3, constructing an improved Mumform-Shah image energy functional model on the basis of the image fusion feature set; s4, setting a space attention adjustment mechanism, and adjusting the weight of a boundary length item; s5, variational optimization is executed to generate an edge response diagram, and a boundary contour is extracted to form a chip packaging structure boundary diagram; s6, fusing the boundary graph and the three-dimensional structure image to construct a boundary graph, and extracting graph structure features; and S7, constructing a packaging quality scoring function, and outputting a defect type, a defect position and a packaging quality scoring value. According to the method, two-dimensional and three-dimensional structural features are fused, and the modeling capability of chip packaging defect identification and scoring is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of semiconductor manufacturing inspection, and particularly to an optimization method for chip package quality inspection based on 2D and 3D composite imaging. Background Art

[0002] In the field of semiconductor manufacturing, chip package quality is an important indicator affecting the reliability and service life of integrated circuits. Especially in the context of the wide application of advanced packaging forms such as high-density packaging and three-dimensional stacked packaging, higher requirements are put forward for the detection accuracy and coverage rate of chip package defects. The currently commonly used package quality inspection technologies mainly rely on two-dimensional visible light images or X-ray images, and identify geometric anomalies or connection defects of package structures such as solder joints, pins, and bonding wires through image processing algorithms. Common image processing methods include edge detection, morphological analysis, template matching, and convolutional neural network classification, etc., which have certain recognition capabilities in scenarios with good lighting conditions and clear structures. However, for non-surface visible defects such as multi-layer structure superposition, slight deformation caused by internal stress, pad occlusion, and solder joint voids, the detection ability is significantly insufficient.

[0003] In order to improve the ability to obtain deep structure information, some studies have introduced three-dimensional structure imaging means, such as structured light scanning, point cloud reconstruction, and laser ranging, etc., to obtain the three-dimensional depth image of the chip package structure, and then combine it with the two-dimensional image to form a detection mode of multi-source information fusion. This 2D and 3D composite detection idea can, to a certain extent, make up for the problem of insufficient defect expression from a single perspective. In the prior art, depth maps and two-dimensional images are often used for pixel-level registration, and then fusion features are generated through simple image superposition, average filtering, or geometric feature stitching, and then input into traditional classification networks or threshold-based rule algorithms for defect classification. However, in the case of complex package structures, irregular boundaries, serious surface reflection or occlusion interference, such fusion methods cannot effectively extract spatial boundary information, and lack the ability to depict the continuity and spatial consistency of package defects, resulting in the risk of false detection and missed detection in the final defect determination.

[0004] In addition, in the existing image modeling mechanism, edge detection operators or graph cut algorithms based on gray-scale changes are generally used as tools for extracting the boundaries of package structures. Such methods are difficult to adapt to the problems of gray-scale non-uniformity, structure density changes, and noise disturbances in package images. The classical Mumford-Shah energy model provides a mathematical optimization framework for image segmentation, which can simultaneously consider image smoothing, boundary accuracy, and image reconstruction consistency, but it is limited in scenarios with serious gray-scale drift or complex structures. The traditional model fails to adjust the boundary term response by combining three-dimensional structure information, so its applicability in the chip package inspection task is insufficient.

[0005] On the other hand, although some methods introduce deep neural networks for defect classification, their training process is highly sensitive to the quantity of the dataset and the annotation accuracy, and the model has poor interpretability, making it difficult to meet the requirements of industrial package detection scenarios that strongly rely on traceability and physical rationality. Existing models generally lack the ability to model the coherence and consistency of the package structure boundary in three-dimensional space. Especially when there are structural defects such as solder joint defects, metal layer offsets, and bonding misalignments, they cannot expand local anomalies into global judgment bases through graph structures or spatial constraints, resulting in fragmented defect boundary determination and fuzzy scoring mechanisms.

[0006] Therefore, how to provide an optimized method for chip package quality detection based on 2D and 3D composite imaging is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0007] An object of the present invention is to propose an optimized method for chip package quality detection based on 2D and 3D composite imaging. The present invention fully integrates spatial registration processing of two-dimensional images and three-dimensional structure images, image energy functional modeling, boundary atlas construction, and graph structure feature analysis methods, and details the process of extracting the package structure boundary through an improved Mumford-Shah model and identifying defect types and scoring package quality based on the graph structure, having the advantages of clear feature expression, accurate boundary modeling, and strong robustness of the scoring mechanism.

[0008] An optimized method for chip package quality detection based on 2D and 3D composite imaging according to an embodiment of the present invention includes the following steps: S1. Collect two-dimensional images and three-dimensional structure images of the chip package area; S2. Perform spatial registration and normalization processing on the two-dimensional images and three-dimensional structure images, extract luminance gradients, depth gradients, and structural curvatures, and generate an image fusion feature set; S3. Build an image energy functional model based on the image fusion feature set. The image energy functional model is an improved Mumford-Shah model, including a regional smoothing term, a reconstruction fidelity term, and a boundary length term; S4. Set a spatial attention adjustment mechanism in the image energy functional model to spatially adjust the weight of the boundary length term according to different structure distributions in the chip package area; S5. Use a variational optimization method to solve the image energy functional model, generate an edge response map, extract the boundary contour of the chip package area, and form a chip package structure boundary map; S6. Fusion the chip package structure boundary map with the three-dimensional structure image, build a boundary atlas of the chip package area, and extract boundary continuity features, boundary consistency features, and structural anomaly features; S7. Construct an encapsulation quality scoring function, taking the boundary continuity feature, boundary consistency feature, and structural anomaly feature as inputs, and generating the defect type, defect location, and encapsulation quality score value of the chip encapsulation area.

[0009] Optionally, the two-dimensional image includes a visible light image or an X-ray image, and the three-dimensional structure image includes a depth image obtained by a point cloud scanning device or a depth sensor.

[0010] Optionally, the regional smoothing term is constructed based on the luminance gradient and depth gradient in the image fusion feature set, and is defined within the chip encapsulation area outside the boundary map.

[0011] Optionally, the reconstruction fidelity term is constructed based on the difference between the pixel values of the two-dimensional image and the output values of the image model, and is defined over the entire chip encapsulation area.

[0012] Optionally, S2 specifically includes: S21. Perform spatial registration on the two-dimensional image and the three-dimensional structure image to establish the correspondence between the two-dimensional image coordinate system and the three-dimensional structure image coordinate system; S22. Normalize the spatially registered two-dimensional image and three-dimensional structure image respectively, and standardize the pixel values in the two-dimensional image and the depth values in the three-dimensional structure image to the interval [0, 1]; S23. Extract the luminance gradient from the normalized two-dimensional image, and extract the depth gradient and structural curvature from the normalized three-dimensional structure image; S24. Construct an image fusion feature set vector at the coordinate position (x, y) , where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the luminance gradient, is the depth gradient, is the structural curvature.

[0013] Optionally, S3 specifically includes: S31. Set the image gray function: ; where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y), is the two-dimensional Gaussian kernel function with a standard deviation of , is the standard deviation of the Gaussian kernel, is the convolution operator, and u(x, y) is the image gray function; S32. Set the boundary weighting function: ; Among them, is the boundary weighting function, is the depth gradient of the three-dimensional structure image at the coordinate (x, y), is the Euclidean norm of the depth gradient, is the structure curvature, is the absolute value of the structure curvature, is the boundary reference weight coefficient, is the adjustment coefficient of the depth guiding factor, is the adjustment coefficient of the structure curvature adjustment function; S33. Construct an image energy functional model: ; Among them, is the image energy functional model, is the two-dimensional domain of the chip packaging area, is the set of boundary curves, is the square of the gradient modulus of the image grayscale function at the coordinate (x, y), and ds is the infinitesimal length on the integral path along the boundary curve, , , are the non-negative weighting coefficients of the region smoothing term, the reconstruction fidelity term, and the boundary length term respectively; S34. The boundary length term is constructed by performing a path integral on the boundary curve set for the boundary weighting function . The boundary weighting function contains a depth guiding factor and a structure curvature adjustment function calculated from the three-dimensional structure image; S35. The image energy functional model takes the image fusion feature set as the input, the image grayscale function as the variational optimization object, the boundary weighting function as the integrand of the boundary integral term, and the image energy functional model serves as the basic structure for generating the edge response map and constructing the chip packaging structure boundary map.

[0014] Optionally, the S4 specifically includes: S41. Set the spatial adjustment function: ; Among them, x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the spatial adjustment reference coefficient, , are the spatial adjustment factors, is the boundary density function of the chip packaging structure boundary at the coordinate (x, y), is the structure complexity function of the chip packaging structure at the coordinate (x, y); S42. Adjust the boundary weighting function in the image energy functional model based on the spatial adjustment function, and set the boundary weighting function after spatial adjustment: ; Among them, is the boundary weighting function, is the boundary weighting function after spatial adjustment, is the spatial adjustment function; S43. Replace the integrand in the boundary length term with the boundary weighting function after spatial adjustment, and update the boundary length term expression to Among them, is the set of boundary curves extracted from the chip packaging area, and ds is the path differential length on the boundary curve.

[0015] Optionally, the specific steps of S5 are as follows: S51. Set the image energy functional model as the objective function for variational optimization: ; Among them, is the image energy functional model, u(x, y) is the image gray function, I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y), is the square of the gradient modulus of the image gray function, is the boundary weighting function after spatial adjustment, is the set of boundary curves, is the two-dimensional domain of the chip packaging area, , , are non-negative weighting coefficients, ds is the differential length of the integration path along the boundary curve, x is the horizontal coordinate of the image, and y is the vertical coordinate of the image; S52. Perform variational optimization on the image energy functional model to obtain an edge response map. The edge response map is a two-dimensional matrix with the same size as the original image, and each pixel position corresponds to a boundary response value; S53. According to the comparison result between the pixel value in the edge response map and the boundary response threshold, extract the boundary contour of the chip packaging area. The boundary contour is composed of a set of pixel coordinate points that meet the response conditions; S54. Represent the pixel coordinate points in the boundary contour as a coordinate sequence Among them is the abscissa of the i-th boundary point, is the ordinate of the i-th boundary point, and i is a positive integer index value; S55. Arrange the coordinate sequence according to the boundary connectivity relationship to construct a boundary map of the chip packaging structure. The boundary map of the chip packaging structure is composed of a set of boundary curves representation

[0016] Optionally, S6 specifically includes: S61. Extract coordinates from the boundary map of the chip packaging structure to obtain a set of boundary coordinates , where is the horizontal coordinate of the i-th boundary point, is the vertical coordinate of the i-th boundary point, N is the number of boundary points, and i is a positive integer index; S62. Map the set of boundary coordinates to the three-dimensional structure image, and obtain the depth value at each boundary point to form a three-dimensional boundary point set , where is the output value of the depth function of the three-dimensional structure image at the coordinate , and is the depth value of the corresponding point; S63. Based on the three-dimensional boundary point set P, construct a boundary map of the chip packaging area, and set the boundary map structure as G=(V,E), where V is the set of nodes and E is the set of edges. The node represents the three-dimensional boundary point , and the edge represents the connection relationship between adjacent nodes; S64. Calculate the boundary continuity feature on the graph G: ; where is the boundary continuity feature, representing the average value of the squares of the spatial Euclidean distances between adjacent three-dimensional boundary points, represents the modulus of the three-dimensional coordinate difference between the node and , and N is the number of boundary points; S65. Calculate the boundary consistency feature on the graph G: ; where is the boundary consistency feature, representing the entropy value of the boundary normal vector direction distribution. K is the number of intervals of direction discretization, is the normalized frequency of the boundary normal vector in the j-th direction interval, and the lower the entropy value, the more concentrated the direction; S66. Calculate the structural anomaly feature on the graph G: ; where is the structural anomaly feature, represents the modulus of the depth gradient of the three-dimensional structure image at the point , is the structural curvature calculated at point .

[0017] Optionally, the S7 specifically includes: S71. Construct an encapsulation quality scoring function: ; where Q is the encapsulation quality score value of the chip encapsulation area, is the boundary continuity feature, is the boundary consistency feature, is the structural anomaly feature, , , are non - negative weighting coefficients of the encapsulation quality scoring function; S72. Based on the boundary point coordinates in the chip encapsulation structure boundary map, the corresponding depth values in the three - dimensional structure image, and the graph structure feature vector, determine the defect type for each boundary point to obtain a defect type label; S73. Associate the spatial coordinates, defect type label of each boundary point with the encapsulation quality score value to generate a result set, and the result set contains the defect positions, defect types, and encapsulation quality score values of each boundary point in the chip encapsulation area.

[0018] The beneficial effects of the present invention are: By constructing an optimized method for chip encapsulation quality detection based on 2D and 3D composite imaging, the present invention realizes accurate boundary modeling and structural feature extraction of the encapsulation area on the basis of spatial registration and fusion of two - dimensional images and three - dimensional structure images. Compared with the existing detection methods that rely on a single image dimension or low - level feature stitching, the present invention adopts an image fusion feature set that combines luminance gradient, depth gradient, and structural curvature, effectively enhancing the discrimination ability between the encapsulation boundary and the internal structure. At the same time, by introducing an improved Mumford - Shah image energy functional model, combining the boundary weighting function with the spatial attention adjustment mechanism, a variational optimization framework including a regional smoothing term, a reconstruction fidelity term, and a boundary length term is constructed, improving the boundary response accuracy for regions with uneven gray levels and positions with sudden changes in structural curvature.

[0019] In addition, the present invention constructs a boundary map of the chip packaging area by fusing the edge response map with the three-dimensional structure image, and further extracts boundary continuity, boundary consistency, and structural anomaly features to construct a graph structure feature vector, enabling the packaging quality assessment to take into account the joint expression of structural integrity and local anomalies. The packaging quality scoring function is based on a weighted linear combination form to form an adjustable parameter model, which can quantitatively evaluate the influence range of different types of structural defects on the scoring value. The identification of defect types is realized through the linkage of the boundary map and the graph structure features, improving the recognition ability of minor defects and spatial asymmetric structural anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, but do not constitute a limitation to the present invention. In the accompanying drawings: Figure 1 is the overall flowchart of an optimized method for chip packaging quality detection based on 2D and 3D composite imaging proposed by the present invention; Figure 2 is a schematic diagram of the construction of an image energy functional model and boundary extraction of an optimized method for chip packaging quality detection based on 2D and 3D composite imaging proposed by the present invention; Figure 3 is a schematic diagram of the construction of a boundary map and the output of packaging quality scoring of an optimized method for chip packaging quality detection based on 2D and 3D composite imaging proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0022] Reference Figures 1-3 , an optimized method for chip packaging quality detection based on 2D and 3D composite imaging, includes the following steps: S1. Collect the two-dimensional image and three-dimensional structure image of the chip packaging area; S2. Perform spatial registration and normalization processing on the two-dimensional image and three-dimensional structure image, extract the luminance gradient, depth gradient, and structural curvature, and generate an image fusion feature set; S3. Construct an image energy functional model based on the image fusion feature set. The image energy functional model is an improved Mumford-Shah model, including a region smoothing term, a reconstruction fidelity term, and a boundary length term; S4. Set a spatial attention adjustment mechanism in the image energy functional model to spatially adjust the weight of the boundary length term according to the different structural distributions in the chip packaging area; S5. Use the variational optimization method to solve the image energy functional model, generate an edge response map, extract the boundary contour of the chip packaging area, and form a chip packaging structure boundary map; S6. Integrate the chip packaging structure boundary map with the three-dimensional structure image to construct a boundary map of the chip packaging area, and extract boundary continuity features, boundary consistency features, and structural anomaly features; S7. Construct a packaging quality scoring function, use the boundary continuity features, boundary consistency features, and structural anomaly features as inputs, and generate the defect type, defect location, and packaging quality score value of the chip packaging area.

[0023] The present invention constructs a two-dimensional image and three-dimensional structure image acquisition process for the chip packaging area, establishes a multi-modal information basis, provides complete input conditions for subsequent image fusion and structural boundary analysis, and enhances the model's ability to obtain packaging information.

[0024] In this embodiment, the two-dimensional image includes a visible light image or an X-ray image, and the three-dimensional structure image includes a depth image obtained by a point cloud scanning device or a depth sensor.

[0025] The present invention clearly defines the source types of the two-dimensional image and the three-dimensional structure image, and uses spatial registration and normalization processing means to ensure the extraction of luminance gradients, depth gradients, and structural curvatures in a unified coordinate system, improving the accuracy of feature expression and the stability of the fusion effect.

[0026] In this embodiment, the regional smoothing term is constructed based on the luminance gradient and depth gradient in the image fusion feature set and is defined in the chip packaging area outside the boundary map.

[0027] The present invention constructs a regional smoothing term based on the luminance gradient and depth gradient and defines its scope of action in the packaging area outside the boundary map, enabling the energy model to continuously model the internal feature changes in the packaging area and enhancing the processing ability for the structural transition area.

[0028] In this embodiment, the reconstruction fidelity term is constructed based on the difference between the pixel values of the two-dimensional image and the output values of the image model and is defined over the entire chip packaging area.

[0029] The present invention constructs a reconstruction fidelity term using the difference between the pixel values of the two-dimensional image and the model output values and defines this term over the entire chip packaging area, effectively maintaining the fidelity of the image model to the original image structure information and improving the overall stability of the model.

[0030] In this embodiment, S2 specifically includes: S21. Perform spatial registration on the two-dimensional image and the three-dimensional structure image to establish the correspondence between the two-dimensional image coordinate system and the three-dimensional structure image coordinate system; S22. Perform normalization processing on the spatially registered two-dimensional image and three-dimensional structure image respectively, and standardize the pixel values in the two-dimensional image and the depth values in the three-dimensional structure image to the interval [0, 1]; S23. Extract the brightness gradient in the normalized two-dimensional image, and extract the depth gradient and structure curvature in the normalized three-dimensional structure image; S24. At the coordinate position (x, y), construct an image fusion feature set vector , where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the brightness gradient, is the depth gradient, is the structure curvature.

[0031] The present invention details the construction process of the image fusion feature set, including spatial registration of two-dimensional and three-dimensional images, normalization processing, brightness and depth gradient calculation, and structure curvature extraction, establishing a clear and executable preprocessing process, which is convenient for application and implementation in the packaging detection task.

[0032] In this embodiment, the specific content of S3 includes: S31. Set the image gray function: ; where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y), is a two-dimensional Gaussian kernel function with a standard deviation of , is the standard deviation of the Gaussian kernel, is the convolution operator, and u(x, y) is the image gray function; S32. Set the boundary weighting function: ; where, is the boundary weighting function, is the depth gradient of the three-dimensional structure image at the coordinate (x, y), is the Euclidean norm of the depth gradient, is the structure curvature, is the absolute value of the structure curvature, is the boundary reference weight coefficient, is the adjustment coefficient of the depth guiding factor, is the adjustment coefficient of the structure curvature adjustment function; S33. Construct an image energy functional model: ; Among them, is the image energy functional model, is the two-dimensional domain of the chip packaging area, is the set of boundary curves, is the square of the gradient modulus of the image grayscale function at the coordinates (x, y), and ds is the differential element length on the integral path along the boundary curve, , , are respectively the non-negative weighting coefficients of the regional smoothing term, the reconstruction fidelity term and the boundary length term; S34, the boundary length term is constructed by performing a path integral on the boundary curve set on the boundary weighting function , and the boundary weighting function contains a depth guiding factor and a structure curvature adjustment function calculated from the three-dimensional structure image; S35, the image energy functional model takes the image fusion feature set as the input, the image grayscale function as the variational optimization object, the boundary weighting function as the integrand of the boundary integral term, and the image energy functional model serves as the basic structure for generating the edge response map and constructing the chip packaging structure boundary map.

[0033] The present invention constructs an improved Mumford-Shah energy functional model by setting the image grayscale function and the boundary weighting function, and introducing a structure curvature adjustment function and a depth guiding factor, enhancing the response ability of the model to three-dimensional boundary changes, and forming the basis for subsequent optimization and solution.

[0034] In this embodiment, the S4 specifically includes: S41, setting the spatial adjustment function: ; where x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the spatial adjustment reference coefficient, , are the spatial adjustment factors, is the boundary density function of the chip packaging structure boundary at the coordinates (x, y), is the structure complexity function of the chip packaging structure at the coordinates (x, y); S42, adjusting the boundary weighting function in the image energy functional model based on the spatial adjustment function, and setting the spatially adjusted boundary weighting function: ; where is the boundary weighting function, is the boundary weighting function after spatial adjustment, is the spatial adjustment function; S43. Replace the integrand in the boundary length term with the spatially adjusted boundary weighting function, and update the boundary length term expression to , where is the set of boundary curves extracted from the chip packaging area, and ds is the path element length on the boundary curve.

[0035] The present invention sets a spatial attention adjustment function and regulates the boundary weighting function, enabling the boundary length term to have an adaptive adjustment ability in different packaging structure distribution regions, thereby improving the expression accuracy of the boundary curve of the model in complex structure regions.

[0036] In this embodiment, the S5 specifically includes: S51. Set the image energy functional model as the objective function for variational optimization: ; where is the image energy functional model, u(x, y) is the image gray function, I(x, y) is the pixel value of the original two-dimensional image at the coordinate (x, y), is the square of the gradient modulus of the image gray function, is the spatially adjusted boundary weighting function, is the set of boundary curves, is the two-dimensional domain of the chip packaging area, , , are non-negative weighting coefficients, ds is the element length of the integration path along the boundary curve, x is the horizontal coordinate of the image, and y is the vertical coordinate of the image; S52. Perform variational optimization on the image energy functional model to obtain an edge response map. The edge response map is a two-dimensional matrix with the same size as the original image, and each pixel position corresponds to a boundary response value; S53. According to the comparison result between the pixel value in the edge response map and the boundary response threshold, extract the boundary contour of the chip packaging area. The boundary contour is composed of a set of pixel coordinate points that meet the response conditions; S54. Represent the pixel coordinate points in the boundary contour as a coordinate sequence , where is the abscissa of the i-th boundary point, is the ordinate of the i-th boundary point, and i is a positive integer index value; S55. Arrange the coordinate sequence according to the boundary connectivity relationship to construct a boundary map of the chip packaging structure. The boundary map of the chip packaging structure is represented by the set of boundary curves .

[0037] The present invention solves the image energy functional model through a variational optimization method, generates an edge response map, extracts the boundary contour of the package structure in combination with the response value, and further constructs the boundary map of the chip package structure, realizing the structural conversion process from the fusion feature to the boundary map.

[0038] In this embodiment, step S6 specifically includes: S61. Extract the coordinates of the boundary map of the chip package structure to obtain a set of boundary coordinates , where is the horizontal coordinate of the i-th boundary point, is the vertical coordinate of the i-th boundary point, N is the number of boundary points, and i is a positive integer index; S62. Map the set of boundary coordinates to the three-dimensional structure image, and obtain the depth value at each boundary point to form a three-dimensional boundary point set , where is the output value of the depth function of the three-dimensional structure image at the coordinate , and is the depth value of the corresponding point; S63. Based on the three-dimensional boundary point set P, construct the boundary map of the chip package area, and set the boundary map structure as G=(V, E), where V is the set of nodes and E is the set of edges. The node represents the three-dimensional boundary point , and the edge represents the connection relationship between adjacent nodes; S64. Calculate the boundary continuity feature on the graph G: ; where is the boundary continuity feature, representing the average value of the squares of the spatial Euclidean distances between adjacent three-dimensional boundary points, represents the modulus of the three-dimensional coordinate difference between the node and , and N is the number of boundary points; S65. Calculate the boundary consistency feature on the graph G: ; where is the boundary consistency feature, representing the entropy value of the distribution of the boundary normal vector directions. The lower the entropy value, the more concentrated the directions; is the normalized frequency of the boundary normal vector in the j-th direction interval, and the lower the entropy value, the more concentrated the directions; S66. Calculate the structural anomaly feature on the graph G: ; where​ is a structural anomaly feature, indicating the magnitude of the depth gradient at a point in the three-dimensional structure image and is the structural curvature calculated at the point is the point where the structural curvature is calculated.

[0039] The present invention performs joint modeling based on a three-dimensional structure image and a chip package structure boundary map, establishes a three-dimensional boundary point set and a map structure, and extracts boundary continuity, boundary consistency, and structural anomaly features from the map to form a quantifiable map structure feature vector, providing support for defect determination.

[0040] In this embodiment, S7 specifically includes: S71. Construct a package quality scoring function: ; where Q is the package quality score value of the chip package area, is the boundary continuity feature, is the boundary consistency feature, is the structural anomaly feature, , , are non-negative weighting coefficients of the package quality scoring function; S72. Based on the boundary point coordinates in the chip package structure boundary map, the corresponding depth values in the three-dimensional structure image, and the map structure feature vector, determine the defect type for each boundary point to obtain a defect type label; S73. Associate the spatial coordinates, defect type label, and package quality score value of each boundary point to generate a result set, where the result set includes the defect location, defect type, and package quality score value of each boundary point in the chip package area.

[0041] The present invention constructs a package quality scoring function, generates a score value using three types of map structure features as input, and labels the defect type and location based on the scoring result and structural information, realizing the linkage expression of quantitative evaluation of chip package quality and spatial defect localization.

[0042] Example 1: To verify the feasibility of the present invention in implementation, the present invention is applied to the chip package inspection task of a certain chip manufacturing enterprise in the fourth quarter of 2024. The packaging production line of this enterprise adopts a hybrid packaging process of high-density flip-chip bonding and multi-layer BGA, and the products are widely used in the fields of consumer electronics and industrial control. Due to the complex packaging structure and the existence of multi-layer stacking, the traditional two-dimensional image-based detection method has poor effects in identifying structural defects such as micro-warpage, pad misalignment, and virtual soldering, with a high missed detection rate, seriously affecting the chip yield.

[0043] In this packaging detection scenario, the detection system is deployed at the detection station of Production Line No. 1 in the factory area, and mainly consists of a dual-mode imaging module, an image processing and analysis module, and a scoring and feedback module. The dual-mode imaging module simultaneously acquires the visible light image and the structured light scanning depth map of the chip packaging area. The image size is 2048×2048 pixels, and the depth accuracy is 0.05 mm. Image registration adopts the method based on SIFT feature point matching and RANSAC correction, and the normalization range is unified to [0, 1]. Subsequently, the brightness gradient, depth gradient, and structural curvature are extracted to construct an image fusion feature set.

[0044] In the image analysis stage, the system calls an image energy functional model based on the improved Mumford-Shah model. This model includes a regional smoothing term, a reconstruction fidelity term, and a boundary length term. The boundary length term introduces a regulation function based on structural curvature and a guiding factor based on depth gradient, enhancing the model's response ability to depth perturbations and sudden changes in the packaging boundary. At the same time, to adapt to the influence of different packaging structure distributions on boundary expression, the system introduces a spatial attention regulation mechanism to adjust the boundary weighting function regionally. The model optimization uses a variational solution algorithm to iterate and converge, and finally outputs an edge response map and a chip packaging structure boundary map.

[0045] The system fuses the boundary map and the depth map to construct a boundary map of the chip packaging area, and extracts boundary continuity, boundary consistency, and structural anomaly features based on the graph structure to generate a three-dimensional structure feature vector. The scoring function outputs the packaging quality score value in a three-term weighted combination manner. The score range is [0, 100], and the defect type and location are identified according to the structural changes and local abnormal points.

[0046] In mid-December 2024, the system detected a total of 800 samples on the production line of a batch of packaged chip products numbered FC-MD422, among which 200 were defect samples manually reviewed and marked. The total detection time of the method of the present invention is 4.3 seconds per chip, and the corresponding average manual detection time is 9.1 seconds per chip. In terms of defect recognition accuracy, the comparison with the manually marked results is shown in Table 1: Table 1 Defect Detection Statistical Table ; In addition, in terms of graph structure scoring, the comparison between the packaging scoring results and the manual quality inspection scoring shows that the average deviation between the scoring value of this method and the manual scoring result is 1.74 points (full score 100), and the scoring consistency is better than that of the traditional deep learning detection model (deviation 4.91 points). The scoring output example is shown in Table 2: Table 2 Chip Scoring and Defect Recognition Example Table ; As can be seen from the data, the method of the present invention demonstrates excellent structure recognition ability, boundary modeling accuracy, and scoring stability in actual production scenarios. It not only significantly outperforms manual inspection and traditional deep models in terms of detection time, but also achieves a substantial improvement in defect recognition accuracy and scoring consistency.

[0047] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes, shall be covered by the protection scope of the present invention.

Claims

1. A chip packaging quality inspection optimization method based on 2D and 3D composite imaging, characterized in that: The steps include: S1, collecting two-dimensional images and three-dimensional structural images of the chip packaging area; S2, performing spatial registration and normalization processing on the two-dimensional image and the three-dimensional structure image, extracting brightness gradient, depth gradient and structure curvature, and generating an image fusion feature set; S3, constructing an image energy functional model based on the image fusion feature set, wherein the image energy functional model is an improved Mumford-Shah model, including a regional smoothing term, a reconstruction fidelity term, and a boundary length term; S4, setting a spatial attention adjustment mechanism in the image energy functional model to spatially adjust the weight of the boundary length term according to different structural distributions in the chip packaging area; S5. Using a variational optimization method to solve the image energy functional model, generate an edge response map, extract the boundary contour of the chip packaging area, and form a chip packaging structure boundary map; S6, fusing the chip packaging structure boundary map with the three-dimensional structure image to construct a boundary map of the chip packaging area, and extracting boundary continuity features, boundary consistency features and structural abnormality features; S7. Construct a packaging quality scoring function, take the boundary continuity feature, boundary consistency feature and structural abnormality feature as input, and generate the defect type, defect location and packaging quality score value of the chip packaging area.

2. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1 is characterized in that: The two-dimensional image includes a visible light image or an X-ray image, and the three-dimensional structure image includes a depth image acquired by a point cloud scanning device or a depth sensor.

3. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1 is characterized in that: The regional smoothing term is constructed based on the brightness gradient and the depth gradient in the image fusion feature set, and is defined within the chip packaging area outside the boundary atlas.

4. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1 is characterized in that: The reconstruction fidelity term is constructed based on the difference between the pixel value of the two-dimensional image and the output value of the image model, and is defined in the entire chip packaging area.

5. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1 is characterized in that: The S2 specifically includes: S21, performing spatial registration on the two-dimensional image and the three-dimensional structure image, and establishing a corresponding relationship between the two-dimensional image coordinate system and the three-dimensional structure image coordinate system; S22, performing normalization processing on the two-dimensional image and the three-dimensional structure image after spatial registration, and normalizing the pixel value in the two-dimensional image and the depth value in the three-dimensional structure image to the interval [0, 1]; S23, extracting a brightness gradient from the normalized two-dimensional image, and extracting a depth gradient and a structural curvature from the normalized three-dimensional structural image; S24. Construct the image fusion feature set vector at the coordinate position (x, y) , where x is the horizontal coordinate of the image and y is the vertical coordinate of the image. is the brightness gradient, is the depth gradient, is the structural curvature.

6. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1 is characterized in that: The S3 specifically includes: S31. Set the image grayscale function: ; Among them, x is the horizontal coordinate of the image, y is the vertical coordinate of the image, I(x,y) is the pixel value of the original two-dimensional image at the coordinate (x,y), The standard deviation is The two-dimensional Gaussian kernel function, is the standard deviation of the Gaussian kernel, is the convolution operator, u(x,y) is the image grayscale function; S32, setting the boundary weighting function: ; in, is the boundary weighting function, is the depth gradient of the three-dimensional structure image at the coordinate (x, y), is the Euclidean norm of the depth gradient, is the structural curvature, is the absolute value of the structural curvature, is the boundary reference weight coefficient, is the adjustment coefficient of the depth guidance factor, is the adjustment coefficient of the structural curvature adjustment function; S33. Construct image energy functional model: ; in, is the image energy functional model, is the two-dimensional domain of the chip packaging area, is the set of boundary curves, is the square of the gradient modulus of the image grayscale function at the coordinate (x, y), ds is the length of the infinitesimal element on the integral path along the boundary curve, , , are the non-negative weighted coefficients of the regional smoothing term, the reconstruction fidelity term and the boundary length term respectively; S34, the boundary length term is passed through the boundary curve set Upper boundary weighted function Path integral construction is performed, and the boundary weighting function includes a depth guidance factor and a structure curvature adjustment function calculated from a three-dimensional structure image; S35. The image energy functional model takes the image fusion feature set as input, the image grayscale function as the variational optimization object, the boundary weighted function as the integrand of the boundary integral term, and the image energy functional model as the basic structure for edge response map generation and chip packaging structure boundary map construction.

7. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1 is characterized in that: The S4 specifically includes: S41, setting space adjustment function: ; Among them, x is the horizontal coordinate of the image, y is the vertical coordinate of the image, is the spatial adjustment reference coefficient, , is the spatial adjustment factor, is the boundary density function of the chip packaging structure boundary at the coordinate (x, y), is the structural complexity function of the chip packaging structure at the coordinate (x, y); S42, adjusting the boundary weighting function in the image energy functional model based on the spatial adjustment function, and setting the boundary weighting function after spatial adjustment: ; in, is the boundary weighting function, is the boundary weighted function after spatial adjustment, is the spatial adjustment function; S43, the integrand in the boundary length term is replaced by the boundary weighted function after spatial adjustment, and the boundary length term expression is updated to ,in, is the set of boundary curves extracted from the chip packaging area, and ds is the path element length on the boundary curve.

8. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1 is characterized in that: The S5 specifically includes: S51, setting the image energy functional model as the objective function of variational optimization; S52, performing variational optimization on the image energy functional model to obtain an edge response map, where the edge response map is a two-dimensional matrix with the same size as the original image, and each pixel position corresponds to a boundary response value; S53, extracting the boundary contour of the chip packaging area according to the comparison result of the pixel value in the edge response map and the boundary response threshold, where the boundary contour is composed of a set of pixel coordinate points that meet the response condition; S54, representing the pixel coordinate points in the boundary contour as a coordinate sequence ,in is the horizontal coordinate of the ith boundary point, is the ordinate of the i-th boundary point, where i is a positive integer index value; S55, the coordinate sequence Arrange according to the boundary connectivity relationship and construct the chip packaging structure boundary map. The chip packaging structure boundary map is composed of a set of boundary curves. express.

9. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1, characterized in that: The S6 specifically includes: S61. Extract the coordinates of the chip packaging structure boundary map to obtain a boundary coordinate set ,in, is the horizontal coordinate of the i-th boundary point, is the longitudinal coordinate of the i-th boundary point, N is the number of boundary points, and i is a positive integer index; S62, the boundary coordinates are collected Mapped into the three-dimensional structure image, at each boundary point Get the depth value , forming a three-dimensional boundary point set ,in, For the three-dimensional structure image in coordinates The output value of the depth function at is the depth value of the corresponding point; S63, construct a boundary map of the chip packaging area based on the three-dimensional boundary point set P, and set the boundary map structure to be a graph G=(V,E), where V is a node set, E is an edge set, and the node Represents a 3D boundary point ,side Indicates the connection relationship between adjacent nodes; S64. Calculate the boundary continuity feature on the graph G: ; in, is the boundary continuity feature, which represents the average value of the square of the spatial Euclidean distance between adjacent three-dimensional boundary points. Representation Node and The modulus of the three-dimensional coordinate difference, N is the number of boundary points; S65. Calculate the boundary consistency feature on graph G: ; in, is the boundary consistency feature, which represents the entropy value of the boundary normal vector direction distribution, K is the number of discrete direction intervals, is the normalized frequency of the boundary normal vector in the jth direction interval. The lower the entropy value, the more concentrated the direction. S66. Calculate the structural anomaly features on graph G: ; in, It is characterized by structural abnormalities. Represents a three-dimensional structure image at point The depth gradient modulus at For point The curvature of the structure calculated at .

10. The chip packaging quality inspection optimization method based on 2D and 3D composite imaging according to claim 1, characterized in that: The S7 specifically includes: S71. Construct packaging quality scoring function: ; Among them, Q is the packaging quality score of the chip packaging area, is the boundary continuity feature, is the boundary consistency feature, It is characterized by structural abnormalities. , , is the non-negative weight coefficient of the packaging quality scoring function; S72, based on the coordinates of the boundary points in the chip packaging structure boundary map, the corresponding depth values ​​in the three-dimensional structure image, and the graph structure feature vector, determine the defect type of each boundary point to obtain a defect type label; S73, associating the spatial coordinates, defect type label and packaging quality score value of each boundary point to generate a result set, wherein the result set includes the defect position, defect type and packaging quality score value of each boundary point in the chip packaging area.

Images