Packaging chip substrate surface warpage detection method and system based on 3D vision

Abstract

The application discloses a kind of based on 3D vision's packaging chip substrate face warping detection method and system, method includes: the real three-dimensional view of chip is obtained by the structured light projection technology in optical triangulation;Based on template matching algorithm, complete chip area is extracted in depth map;Depth map is converted into three-dimensional point cloud data using inverse projection conversion;The point cloud data obtained is filtered, and noise point not belonging to substrate surface is removed;Multiple regions of interest are set on substrate surface, and ROI meeting conditions are screened and retained;RANSAC random sampling consistency algorithm is used to plane fitting reserved ROI to generate reference surface;The maximum and minimum perpendicular distance of each point on substrate surface to reference surface is calculated, and combined with set threshold value, warping is quantitatively judged, to determine whether chip exists warping deformation.The application can effectively extract the warping parameter of single chip, realize the automatic determination of chip warping state.

Description

Technical Field

[0001] This invention relates to the field of chip inspection technology, and in particular to a method and system for detecting the warpage of a packaged chip substrate based on 3D vision. Background Technology

[0002] In the post-Moore's Law era, the development of packaging technologies such as 2.5D / 3D integration and advanced packaging has driven chip substrates towards "high density, thinness, and multi-material stacking." During the packaging process, substrates are prone to slight warping due to factors such as mismatched thermal expansion coefficients of materials and residual stress from high-temperature processes. Such warping (even at the micrometer level) can directly lead to defects such as chip stacking misalignment and solder joint failure. Therefore, it is necessary to perform "high-precision, full-field, and non-destructive" warping detection on the substrate surface, as traditional low-precision, localized detection methods are no longer suitable.

[0003] Meanwhile, electronic devices are developing towards miniaturization and high performance, chip packaging density is constantly increasing, the difficulty of matching the thermal expansion coefficient of the substrate is increasing, and the warping problem is becoming more and more prominent. This makes it difficult for traditional low-precision detection methods to meet the needs, and high-precision warping detection technology is urgently needed.

[0004] In the field of chip substrate warpage detection in advanced packaging technology, existing technical solutions still have several shortcomings in complex scenarios: the statistical distance or density filtering methods used in existing solutions have limited adaptability to mixed noise and are difficult to filter out different types of noise; secondly, traditional methods that rely on least squares fitting of the plane are easily affected by outliers, and the fitting accuracy and stability are difficult to guarantee in scenarios with insufficient point cloud purity; the denoising method that relies on multi-scale grid interpolation has a weak ability to suppress residual interference points and reflective noise, and it is difficult to effectively distinguish between the real plane and outliers. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for detecting the warpage of a packaged chip substrate based on 3D vision, which improves the accuracy, efficiency and stability of chip substrate warpage detection and effectively solves the detection error problem caused by interference structures.

[0006] The technical solution to achieve the objective of this invention is: a method for detecting the warpage of a packaged chip substrate based on 3D vision, comprising:

[0007] Obtain chip depth map and reference template, and acquire real three-dimensional view of chip based on structured light projection technology of optical triangulation;

[0008] The edges between the template and the image to be detected are extracted using Canny edge detection, and geometric contours are extracted using Hough transform. The complete chip region is located using a template matching algorithm, and erroneous contour points are removed after RANSAC verification.

[0009] Inverse projection transformation is used to convert the depth map into 3D point cloud data. Based on the camera intrinsic and extrinsic parameter matrices, the depth map pixel coordinates are converted into 3D coordinates in the world coordinate system.

[0010] The 3D point cloud data is filtered to remove interfering structures such as solder balls and pins, while retaining the effective information of the substrate surface.

[0011] Multiple regions of interest (ROIs) are set on the substrate surface. Interference points remaining in the ROIs are removed by dynamic weight sampling. The percentage of valid points is calculated and qualified ROIs are selected according to the threshold.

[0012] Plane fitting was performed on the filtered point cloud data. Bayesian M estimation was used to adaptively suppress point cloud noise. Robust principal component analysis was combined to separate the low-rank and sparse components of the point cloud to obtain a purer planar point set. RANSAC was used to perform plane fitting on the processed point cloud data to generate a reference surface.

[0013] Calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference surface, and combine this with a set threshold to quantify the warpage and determine whether the chip is bent or deformed.

[0014] Furthermore, the structured light projection technology based on optical triangulation to acquire a true three-dimensional view of the chip specifically involves: using a 3D vision system to emit structured light of a specific pattern; the structured light is reflected after being projected onto the surface of the chip substrate; the reflected light signal is captured by a sensor; and the three-dimensional coordinates of each point on the chip surface are calculated based on the principle of optical triangulation to form a true three-dimensional view.

[0015] Furthermore, the steps involve extracting the edges between the template and the image to be detected using Canny edge detection, extracting the geometric contour using Hough transform, locating the complete chip region using a template matching algorithm, and removing incorrectly matched contour points using RANSAC verification. The specific steps are as follows:

[0016] The depth maps of the reference template and the chip under test are converted to grayscale. The Canny edge detection algorithm is used to extract the chip edges. Appropriate high and low thresholds are set to retain clear edge information. Specifically, a high threshold H is set. threshold Low threshold L threshold To preserve the chip's bezel outline;

[0017] The extracted edge map is subjected to Hough transform to identify the rectangular boundary geometric contour of the chip and obtain the coordinate information of the contour points.

[0018] Calculate the shape context descriptor of the contour points, and use the shape context descriptor to measure the local feature similarity between the template contour and the contour of the image to be detected;

[0019] Template matching is performed based on local feature similarity to initially locate the chip region;

[0020] The RANSAC algorithm is used to verify the matching results, remove the contour points of incorrect matches, and select the optimal matching area to achieve accurate positioning of the complete chip area.

[0021] Furthermore, the method of using inverse projection transformation to convert the depth map into 3D point cloud data specifically involves: based on the camera intrinsic and extrinsic parameter matrices, combining the two-dimensional coordinates of each pixel in the depth map with its corresponding depth value, and calculating the 3D coordinates of the pixel in the world coordinate system using the inverse projection transformation formula. The set of all 3D coordinate points constitutes the 3D point cloud data.

[0022]

[0023] in, Represents the two-dimensional coordinates of a pixel in the depth map; This represents the depth value corresponding to a pixel; Indicates the coordinates of the camera's principal point (internal parameter); This indicates the camera's focal length (internal parameter) in the x and y directions. This represents the camera rotation matrix (external parameter); This represents the camera translation vector (extrinsic parameter); This represents the coordinates of a three-dimensional point in the world coordinate system.

[0024] Furthermore, the filtering process for the three-dimensional point cloud data to remove interfering structures such as solder balls and pins, while retaining the effective information of the substrate surface, specifically involves using a combination of statistical filtering and radius filtering. First, statistical filtering removes isolated noise points that deviate from the average distance of the point cloud by more than a set threshold. Then, radius filtering removes the point cloud data corresponding to small-sized interfering structures such as solder balls and pins, while retaining the effective point cloud of the main area of ​​the substrate surface.

[0025] Statistical filtering: Take 50-100 neighboring points for each point, calculate the average distance between the neighboring points, and remove isolated noise points that deviate from the average distance by more than 2-3 times the standard deviation.

[0026] Radius filtering: Set the search radius R search Point clouds with fewer than 8-12 neighboring points (corresponding to interfering structures such as solder balls and pins) are removed, and the effective point clouds on the substrate surface are retained.

[0027] The statistical filtering operation is as follows:

[0028] Calculate a single point Average neighborhood distance:

[0029]

[0030] Calculate the standard deviation of the neighborhood distances of all points:

[0031]

[0032] Noise point judgment criteria (if met, it will be removed):

[0033]

[0034] Where: k is the number of neighboring points (value 50-100); For point The j-th neighboring point; Where N is the Euclidean distance; N is the total number of points in the point cloud; This represents the mean of the neighborhood average distances; Indicates a multiple of the standard deviation;

[0035] The radius filtering operation is as follows:

[0036] For point If the number of points in its neighborhood meets the following conditions, it will be removed (corresponding to interference such as solder balls and pins):

[0037]

[0038] Where: r is the search radius; m is the minimum number of neighboring points, with a value of 8-12; This indicates the number of neighboring points that meet the statistical conditions.

[0039] Furthermore, the step of setting multiple ROIs on the substrate surface, removing residual interference points in the ROIs through dynamic weighted sampling, calculating the proportion of valid points, and filtering qualified ROIs according to a threshold is specifically as follows:

[0040] Based on the size and shape of the chip substrate, multiple rectangular ROIs are uniformly set on the substrate surface to ensure that the ROIs cover the clean areas of the substrate surface that do not contain interference information such as pins and solder balls;

[0041] Dynamic weight sampling is performed on the point cloud data within each ROI. Points within the substrate plane are assigned high weights, while interference points such as residual burrs, solder balls, and leads are assigned low weights. Low-weight interference points are then removed to obtain high-purity points within the substrate plane. The dynamic weight calculation is as follows:

[0042] Calculate the consistency weight of the normal vector:

[0043] Calculate the average normal vector of all points within the ROI:

[0044]

[0045] Where: M is the total number of point clouds within the ROI; Let be the unit normal vector of the j-th point, satisfying ;

[0046] Based on the dot product to measure directional similarity, calculate the consistency weight of the normal vector:

[0047]

[0048] in This represents the dot product of the average normal vector and the normal vector of the i-th point, with a range of values. The closer it is to 1, the more consistent the directions are; Indicates the maximum permissible normal vector deviation angle; if This indicates that the normal vector deviation is too large. This was identified as a feature of interference points.

[0049] Calculate curvature weights:

[0050]

[0051] in: This represents the actual curvature value at the i-th point. Indicates the maximum permissible curvature threshold; if This indicates that the curvature is too large. This was identified as a feature of interference points.

[0052] The final weight values ​​are determined by the consistency of the point cloud's normal vectors and its curvature:

[0053]

[0054] in: This represents the final weight of the i-th point; , The normal vector weight coefficients and curvature weight coefficients represent the following: ; This represents the consistency weight of the normal vector at the i-th point; This represents the curvature weight at the i-th point.

[0055] Calculate the ratio of the number of high-weight points within each ROI to the total number of points in the ROI to obtain the percentage of effective points;

[0056] Set a threshold for the percentage of valid points, retain ROIs with a percentage of valid points greater than or equal to the threshold as qualified ROIs, and remove ROIs with a percentage of valid points lower than the threshold;

[0057] Specifically, the ROI setting and screening adopts a partitioning strategy for large-size substrates: 1) Divide the substrate into 16-25 rectangular ROIs evenly according to the substrate size to ensure coverage of the four corners and the central area of ​​the substrate; 2) Set the effective point ratio threshold to ≥80%, and after screening qualified ROIs, fit a local plane for each qualified ROI separately, and then obtain the overall reference plane of the substrate through global splicing to avoid edge fitting errors of large-size substrates.

[0058] Furthermore, Bayesian M-estimation (BME) is used to adaptively suppress point cloud noise, and robust principal component analysis (RPCA) is used to separate the low-rank and sparse components of the point cloud to obtain a purer planar point set. The processed point cloud data is then fitted with RANSAC to generate a reference surface. The specific steps are as follows:

[0059] Bayesian M-estimation is used to process point cloud data within the qualified ROI. The estimation parameters are adaptively adjusted according to the statistical characteristics of the point cloud data to suppress point cloud noise caused by factors such as uneven reflection on the chip surface.

[0060] Based on the Gaussian distribution assumption of point clouds, the parameters are estimated. Maximize the posterior probability:

[0061]

[0062] Where the likelihood function A robust loss function is used (to suppress reflection unevenness noise):

[0063]

[0064] Where: X represents point cloud data; For parameters to be estimated (such as plane model parameters); The parameter prior distribution is used (Gaussian prior is adopted); Let be the coordinate vector of the i-th point cloud data point; For parameter-based The coordinate vector of the predicted point; A robust loss function is used (emphasizing Huber loss to balance Gaussian distribution and outlier suppression). denoted as the standard deviation of the neighborhood distance of a point.

[0065] Robust principal component analysis (RPCA) is performed on the denoised point cloud data to decompose the point cloud data into low-rank components and sparse components. The low-rank components correspond to the real substrate plane, and the sparse components correspond to the remaining noise and outliers. The L1 norm is used to separate the low-rank components and sparse components to obtain a purer planar point set.

[0066] Robust principal component analysis specifically includes:

[0067] Construct a data matrix from the filtered point cloud data (N 3D points) within the ROI;

[0068]

[0069] in Let i be the three-dimensional coordinates (column vector form) of the world coordinate system of the i-th point.

[0070] The data matrix X is decomposed into a low-rank component L (corresponding to the actual substrate plane) and a sparse component S (corresponding to outliers / residual interference). The optimization objective is:

[0071]

[0072] in The nuclear norm (sum of singular values) of matrix L is used to constrain the low rank of L (the three-dimensional data of a planar point cloud can be characterized by 1-2 principal components, and the minimum nuclear norm is equivalent to the minimum rank).

[0073] Let L1 be the L1 norm of matrix S (the sum of the absolute values ​​of all elements), used to constrain the sparsity of S (outliers are few and most elements are 0).

[0074] Indicates the balance parameter (adaptive point cloud quantity);

[0075] This represents the low-rank component matrix (each column corresponds to the coordinates of an ideal point on the substrate surface).

[0076] This represents the sparse component matrix (each column corresponds to the deviation of the outlier from the ideal plane).

[0077] After decomposition, the column vector of the low-rank component L is the pure substrate surface point cloud after removing outliers.

[0078] The RANSAC method is used to fit the plane to obtain the reference plane. A robust loss function is introduced to reduce the impact of outliers on the fitting results. The reference plane is iteratively optimized through a multi-scale refinement strategy. Specifically, at a large scale, a global coarse fit is performed using RANSAC+Tukey loss to obtain the fitted plane P1. Then, at a small scale, a local fine fit is performed on the interior points of P1 to obtain the plane P2. Through multiple iterations, if the angle between the normal vectors of two adjacent planes is less than a threshold, the iteration is terminated, which ultimately improves the accuracy of the plane fitting.

[0079] Furthermore, the maximum and minimum vertical distances from each point on the substrate surface to the reference surface are calculated, and the warpage is quantified by setting a threshold to determine whether the chip has bending deformation. Specifically:

[0080] The equation of the reference plane is obtained based on the plane fitting results, and the vertical distance from each effective point in the substrate surface to the reference plane is calculated.

[0081]

[0082] in: Represents the vertical distance from the i-th effective point on the substrate surface to the reference surface, in μm; The parameters of the datum plane equation are: ; This represents the three-dimensional coordinates of the i-th point in the world coordinate system.

[0083] The maximum and minimum values ​​among all vertical distances are counted, and the difference between the maximum and minimum values ​​is used as the quantified value of the warpage of the chip substrate surface.

[0084]

[0085] in: This represents the maximum vertical distance from all valid points to the reference plane; This represents the minimum vertical distance from all valid points to the reference plane; W represents the quantified value of warpage, in μm.

[0086] A warpage threshold is set. If the quantized warpage value is less than or equal to the threshold, the chip is determined to have no obvious warpage deformation; if the quantized warpage value is greater than the threshold, the chip is determined to have warpage deformation.

[0087] A 3D vision-based system for detecting the warpage of a packaged chip substrate, used to implement the above method, the system comprising:

[0088] The 3D view acquisition module is used to acquire chip depth maps and reference templates, and acquires the chip's true 3D view based on structured light projection technology using optical triangulation.

[0089] The chip region localization module is used to extract the edges between the reference template and the image to be detected through Canny edge detection, extract the geometric contours by combining Hough transform, locate the complete chip region using template matching algorithm, and remove erroneous contour points by RANSAC verification.

[0090] The data format conversion module is used to convert depth maps into 3D point cloud data using inverse projection transformation;

[0091] The point cloud filtering and denoising module is used to filter 3D point cloud data, remove noise points, and retain effective information of the substrate surface.

[0092] The ROI setting and filtering module is used to set multiple ROIs on the substrate surface, remove residual interference points in the ROIs through dynamic weight sampling, calculate the proportion of effective points, and filter qualified ROIs according to the threshold.

[0093] The plane fitting module is used to perform plane fitting on the filtered point cloud data. It adopts Bayesian M estimation to adaptively suppress point cloud noise and combines robust principal component analysis to separate the low-rank components and sparse components of the point cloud to obtain a purer plane point set. The processed point cloud data is then fitted with RANSAC to generate a reference surface.

[0094] The warpage quantification and determination module is used to calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference surface, and combine the set threshold to quantify the warpage and determine whether the chip has bending deformation.

[0095] The results output module is used to output the test results, including chip warpage parameters and deformation state judgment conclusions.

[0096] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention integrates the chip region localization scheme of Canny edge detection, Hough transform, shape context descriptor and RANSAC verification, which greatly improves the accuracy of chip region localization and effectively avoids the positioning error caused by edge blurring and interference structure; (2) The combination of statistical filtering and radius filtering is used to denoise the chip region, and the multi-level noise suppression strategy of dynamic weight sampling and Bayesian M estimation (BME) and robust principal component analysis (RPCA) can efficiently remove various interference factors such as solder balls, pins and uneven reflection, and obtain high purity substrate surface point cloud data; (3) The plane fitting method of RANSAC + multi-scale refinement is used to improve the fitting accuracy of the reference plane. Combined with the warp measurement method of the maximum and minimum vertical distance difference, the high-precision warp detection at the micron level is realized, which meets the detection requirements of the packaging field; (4) The entire detection process is highly automated and efficient. It is a non-contact detection method that will not damage the chip and is suitable for industrial online detection scenarios. Attached Figure Description

[0097] Figure 1 This is an overall flowchart of the 3D vision-based method for detecting the warpage of a packaged chip substrate according to the present invention.

[0098] Figure 2 This is a schematic diagram of the chip area positioning process;

[0099] Figure 3This is a schematic diagram of the point cloud filtering and ROI selection process;

[0100] Figure 4 This is a schematic diagram of the plane fitting process;

[0101] Figure 5 A visualization of the chip's original point cloud;

[0102] Figure 6 A visualization of the chip's original point cloud (showing noise data);

[0103] Figure 7 A visualization of the substrate surface after noise removal from the chip point cloud data;

[0104] Figure 8 The image shows the point cloud of the chip after fitting the plane (the red part is the reference plane). Detailed Implementation

[0105] To address the shortcomings of existing technologies, this invention employs multi-stage technological innovations, utilizing an algorithm combining statistical filtering and radius filtering to more flexibly handle different types of noise interference. A fitting method based on ROI clean point clouds and RANSAC random sampling reduces the impact of outliers, while multi-level optimization further improves the accuracy of the baseline fitting. A combined scheme of Dynamic Weighted Sampling (DWS), Bayesian M-estimation (BME), and RPCA efficiently removes residual interference points and suppresses reflective noise, while accurately separating the real plane from outliers. The multi-scale refinement optimization logic, through large-scale initial fitting and small-scale iterative optimization, achieves a gradual improvement in fitting accuracy. The overall solution is more suitable for advanced packaging chip substrate warpage detection scenarios characterized by "high interference, high accuracy requirements, and strong online detection needs," meeting the core requirements of packaging processes for "high precision, high efficiency, and non-destructive testing," thus becoming one of the effective methods for substrate warpage detection. The method of this invention can effectively extract the warpage parameters of a single chip, realize the automatic determination of the chip warpage state, and has high-precision and high-efficiency detection performance. It is suitable for industrial scenarios such as semiconductor packaged chip inspection, precision electronic component assembly quality assessment, precision component surface inspection, and high-precision PCB manufacturing process monitoring.

[0106] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. A detailed description of the 3D vision-based method and system for detecting the warpage of a packaged chip substrate is provided.

[0107] The 3D vision-based packaging chip substrate warpage detection system of this embodiment includes a three-dimensional view acquisition module, a chip region positioning module, a data format conversion module, a point cloud filtering and denoising module, a ROI setting and filtering module, a plane fitting module, a warpage quantification and judgment module, and a result output module. Specifically: the three-dimensional view acquisition module uses a 3D vision sensor based on optical triangulation, emitting structured light; the camera intrinsic parameter matrix is ​​determined through calibration experiments, and the extrinsic parameter matrix is ​​obtained through calibration of the relative position of the camera and the chip; the chip region positioning module, data format conversion module, point cloud filtering and denoising module, ROI setting and filtering module, plane fitting module, and warpage quantification and judgment module are implemented through software programming and run on an industrial control computer; the result output module includes a display and a data storage unit for real-time display of detection results and storage of detection data.

[0108] Combination Figures 1-4 The 3D vision-based method for detecting the warpage of a packaged chip substrate in this embodiment includes the following steps:

[0109] Step 1: Obtain the chip depth map and reference template. Using structured light projection technology based on optical triangulation, acquire a true 3D view of the chip. Fix the packaged chip to be inspected on the inspection platform, ensuring the chip substrate surface is flat and unobstructed. Activate the 3D view acquisition module. The 3D vision sensor emits structured light onto the chip substrate surface. The structured light is reflected by the substrate surface and captured by the sensor. Based on the principle of optical triangulation, calculate the 3D coordinates of each point on the chip surface according to the angle difference between the emitted and reflected light, forming a true 3D view of the chip. Simultaneously, import the reference template for this chip model. The reference template is a chip depth map without warping defects.

[0110] Step 2: Extract the edges between the template and the image to be detected using Canny edge detection, combine this with Hough transform to extract the geometric contours, and use a template matching algorithm to locate the complete chip region. Verify with RANSAC to remove incorrectly matched contour points. Perform grayscale processing on the depth maps of the reference template and the chip to be detected, and use the Canny edge detection algorithm to extract edges, setting a high threshold H. threshold Low threshold L thresholdThe process preserves clear chip edge information; Hough transform is applied to the extracted edges, and the accumulator threshold in the Hough space is set to 100 to identify the rectangular geometric contour of the chip and obtain the coordinate data of the contour points; the shape context descriptor of each contour point is calculated, and multiple angle and distance intervals are divided in polar coordinates with the contour point as the center, and the number of contour points in each interval is counted to form a shape context descriptor; the distance between the shape context descriptors of the template contour and the contour of the image to be detected is calculated to measure the local feature similarity, and template matching is performed based on the local feature similarity to initially locate the chip region; the RANSAC algorithm is used to verify the matching results, such as setting the interior point threshold to 2 pixels, removing incorrectly matched contour points, and selecting the optimal matching region to achieve accurate positioning of the complete chip region, with the positioning error controlled within ±1 pixel.

[0111] Step 3: Convert the depth map into 3D point cloud data using inverse projection transformation. Based on the intrinsic parameter matrix (focal length, principal point coordinates, etc.) and extrinsic parameter matrix (rotation matrix, translation vector, etc.) of the 3D vision sensor, combine the (u, v) 2D coordinates of each pixel in the depth map with the corresponding depth value z to obtain the chip's point cloud map, such as... Figure 5 As shown.

[0112] Step 4: The initial 3D point cloud map carries a large amount of noisy data, such as solder balls, pins, etc. Figure 6 As shown, the denoising process mainly involves filtering the 3D point cloud data to remove interfering structures such as solder balls and pins, while retaining the effective information of the substrate surface. A statistical filtering algorithm is used for initial denoising of the 3D point cloud data, such as setting the number of neighboring points to 50 and the standard deviation to 2.0, to remove isolated noise points whose distance from the average distance of neighboring points exceeds 2 times the standard deviation. A radius filtering algorithm is then used to further remove interfering structures, with the search radius set to R. search The minimum number of neighboring points is 10. Point cloud data corresponding to small-sized interfering structures such as solder balls and leads are removed, and the effective point cloud of the main area of ​​the substrate surface is retained. The processed point cloud map is shown below. Figure 7 As shown.

[0113] Step 5: Set multiple ROIs on the substrate surface. Use dynamic weight sampling to remove residual interference points in the ROIs, calculate the effective point ratio, and filter qualified ROIs according to the threshold. Based on the size of the chip substrate surface, set N ROIs evenly on the substrate surface to ensure coverage of the entire substrate surface. Perform dynamic weight sampling on the point cloud data in each ROI. Based on the normal vector and curvature characteristics of the point cloud, assign high weight (weight value of 1.0) to points in the substrate surface (with consistent normal vectors and small curvature), and assign low weight (e.g., weight value of 0.1) to residual burrs, solder balls, pins, and other interference points (with disordered normal vectors and large curvature). Remove interference points with weight values ​​lower than 0.5 to obtain high purity points in the substrate surface. Calculate the ratio of the number of high-weight points in each ROI to the total number of points in the ROI to obtain the effective point ratio. Set an effective point ratio threshold and retain ROIs with an effective point ratio greater than the threshold as qualified ROIs. In this embodiment, all ROIs are qualified ROIs.

[0114] Step 6: Adaptive suppression of point cloud noise is achieved using Bayesian M-estimation (BME). Robust Principal Component Analysis (RPCA) is used to separate the low-rank and sparse components of the point cloud. BME is applied to the point cloud data within the qualified ROI. Based on the Gaussian distribution characteristics of the point cloud data, the estimation parameters are adaptively adjusted to suppress point cloud noise caused by uneven reflection on the chip surface. Robust Principal Component Analysis (RPCA) is then performed on the denoised point cloud data to construct the covariance matrix. Principal components are obtained through eigenvalue decomposition, decomposing the point cloud data into low-rank components (corresponding to the actual substrate plane) and sparse components (corresponding to residual noise and outliers). L1 grading is then applied. Norm separation separates low-rank and sparse components, yielding a pure planar point set. RANSAC is used for planar fitting to obtain a reference surface. A robust loss function is introduced to reduce the impact of outliers on the fitting results. A multi-scale refinement strategy iteratively optimizes the reference surface: first, RANSAC+Tukey loss is used for global coarse fitting at a large scale, then local fine fitting is performed at a small scale. Through multiple iterations, the accuracy of the planar fitting is ultimately improved, resulting in the fitted reference surface as shown below. Figure 8 As shown in the red area.

[0115] Step 7: Calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference plane. Combine this with a set threshold to quantify the warpage and determine if the chip has bending deformation. Obtain the equation of the reference plane based on the plane fitting results. Calculate the vertical distance from each effective point on the substrate surface to the reference plane. Statistically find the maximum and minimum values ​​among all vertical distances to calculate the quantified warpage value. Based on the packaging requirements of this chip model, set the warpage threshold to W. threshold .

[0116] Step 8: Display the test results in real time through the display of the result output module, including chip warpage parameters and deformation state judgment conclusions; at the same time, store the test data (including 3D point cloud data, reference surface parameters, warpage values, judgment results, etc.) to the data storage unit for subsequent traceability and analysis.

Claims

1. A method for detecting the warpage of a packaged chip substrate based on 3D vision, characterized in that, Includes the following steps: Obtain chip depth map and reference template, and acquire real three-dimensional view of chip based on structured light projection technology of optical triangulation; The edges between the baseline template and the image to be detected are extracted by Canny edge detection, and the geometric contours are extracted by Hough transform. The complete chip area is located by template matching algorithm, and erroneous contour points are removed after RANSAC verification. Inverse projection transformation is used to convert the depth map into 3D point cloud data. Based on the camera intrinsic and extrinsic parameter matrices, the depth map coordinates are converted into 3D coordinates in the world coordinate system. The 3D point cloud data is filtered to remove noise points and retain the effective information of the substrate surface. Multiple Regions of Interest (ROIs) are defined on the substrate surface. Dynamic weighted sampling is used to remove residual interference points from each ROI. The percentage of valid points is calculated, and qualified ROIs are selected based on a threshold. N ROIs are evenly distributed across the substrate surface according to its dimensions, ensuring full coverage. Dynamic weighted sampling is performed on the point cloud data within each ROI. Based on the normal vector and curvature characteristics of the point cloud, high weights are assigned to points within the substrate surface, while low weights are assigned to residual burrs, solder balls, and pin interference points. Interference points with weight values ​​below 0.5 are removed, resulting in highly pure points within the substrate surface. The ratio of the number of high-weight points within each ROI to the total number of points in the ROI is calculated to obtain the percentage of valid points. Set a threshold for the percentage of valid points, and retain ROIs with a percentage of valid points greater than the threshold as qualified ROIs; Plane fitting was performed using the filtered point cloud data. Bayesian M-estimation was used to adaptively suppress point cloud noise, and robust principal component analysis was combined to separate low-rank and sparse components of the point cloud. Bayesian M-estimation was used to process the point cloud data within the qualified ROI. Based on the Gaussian distribution characteristics of the point cloud data, the estimation parameters were adaptively adjusted to suppress point cloud noise caused by uneven reflection on the chip surface. Robust principal component analysis was performed on the denoised point cloud data to construct the covariance matrix of the point cloud data. Principal components were obtained through eigenvalue decomposition, decomposing the point cloud data into low-rank and sparse components. The L1 norm was used to separate the low-rank and sparse components, resulting in a clean planar point set. The RANSAC method was used to fit the plane to obtain the reference surface. A robust loss function was introduced to reduce the impact of outliers on the fitting results. The reference surface was iteratively optimized through a multi-scale refinement strategy. First, RANSAC+Tukey loss was used for global coarse fitting at a large scale, and then local fine fitting was used at a small scale. Through multiple iterations, the accuracy of the plane fitting was finally improved, and the fitted reference surface was obtained. Calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference surface, and combine this with a set threshold to quantify the warpage and determine whether the chip has warpage deformation.

2. The method for detecting the warpage of a packaged chip substrate based on 3D vision according to claim 1, characterized in that, The structured light projection technology based on optical triangulation acquires a true three-dimensional view of the chip. Specifically, a 3D vision system emits structured light, which is reflected after being projected onto the chip substrate. The reflected light signal is captured by a sensor, and the three-dimensional coordinates of each point on the chip surface are calculated based on the principle of optical triangulation to form a true three-dimensional view.

3. The method for detecting the warpage of a packaged chip substrate based on 3D vision according to claim 1, characterized in that, Edges between the baseline template and the image to be detected are extracted using Canny edge detection, and geometric contours are extracted using Hough transform. A template matching algorithm is then used to locate the complete chip region. Incorrectly matched contour points are removed after RANSAC verification. The specific steps are as follows: The depth maps of the reference template and the chip under test are converted to grayscale. The Canny edge detection algorithm is used to extract the chip edges, and high and low thresholds are set to retain clear edge information. The extracted edge map is subjected to Hough transform to identify the rectangular boundary geometric contour of the chip and obtain the coordinate information of the contour points. Calculate the shape context descriptor of the contour points, and use the shape context descriptor to measure the local feature similarity between the template contour and the contour of the image to be detected; Template matching is performed based on local feature similarity to initially locate the chip region; The RANSAC algorithm is used to verify the matching results, remove the contour points with incorrect matches, and select the optimal matching area to achieve accurate positioning of the complete chip area.

4. The method for detecting the warpage of a packaged chip substrate based on 3D vision according to claim 1, characterized in that, The depth map is converted into 3D point cloud data using inverse projection transformation. Specifically, based on the camera intrinsic and extrinsic parameter matrices, the two-dimensional coordinates of each pixel in the depth map are combined with the corresponding depth value. The three-dimensional coordinates of the pixel in the world coordinate system are calculated using the inverse projection transformation formula. The set of all three-dimensional coordinate points constitutes the 3D point cloud data. ； in, Represents the two-dimensional coordinates of a pixel in the depth map; This represents the depth value corresponding to a pixel; Indicates the coordinates of the camera's principal point; This indicates the camera's focal length in the x and y directions; Represents the camera rotation matrix; Represents the camera translation vector; Represents the coordinates of a three-dimensional point in the world coordinate system.

5. The method for detecting the warpage of a packaged chip substrate based on 3D vision according to claim 1, characterized in that, The 3D point cloud data is filtered to remove noise points and retain the effective information of the substrate surface. Specifically, a combination of statistical filtering and radius filtering is used. First, statistical filtering removes isolated noise points that deviate from the average distance of the point cloud by more than a set threshold. Then, radius filtering removes the point cloud data corresponding to interference structures such as solder balls and pins, retaining the effective point cloud of the main area of ​​the substrate surface.

6. The method for detecting the warpage of a packaged chip substrate based on 3D vision according to claim 5, characterized in that, The statistical filtering operation is as follows: Calculate a single point average neighborhood distance : ； Calculate the standard deviation of the neighborhood distances of all points. : ； The following criteria are used to determine noise points; points that meet these criteria will be removed: ； Where: k is the number of neighboring points; For point The j-th neighboring point; Represents the Euclidean distance; N is the total number of points in the point cloud; This represents the mean of the neighborhood average distances; Indicates a multiple of the standard deviation; The radius filtering operation is as follows: For point If the number of points in its neighborhood meets the following condition, it will be removed: ； Where: r is the search radius; m is the number of minimum neighboring points; This indicates the number of neighboring points that meet the statistical conditions.

7. The method for detecting the warpage of a packaged chip substrate based on 3D vision according to claim 1, characterized in that, Multiple regions of interest (ROIs) are defined on the substrate surface. Interference points remaining in the ROIs are removed using dynamic weighted sampling. The percentage of valid points is calculated, and qualified ROIs are selected based on a threshold. Specifically: Based on the size and shape of the chip substrate, multiple rectangular ROIs are uniformly set on the substrate surface to ensure that the ROIs cover the clean areas of the substrate surface that do not contain interference information such as pins and solder balls; Dynamic weight sampling is performed on the point cloud data within each ROI. Points within the substrate plane are assigned high weights, while residual noise points are assigned low weights. Low-weight noise points are then removed to obtain the points within the substrate plane. The dynamic weight calculation is as follows: Calculate the average normal vector of all points within the ROI. : ； Where M is the total number of point clouds within the ROI; Let be the unit normal vector of the j-th point, satisfying ; Based on the dot product to measure directional similarity, the consistency weight of the normal vector is calculated. : ； in This represents the dot product of the average normal vector and the normal vector of the i-th point, with a range of values. The closer it is to 1, the more consistent the directions are; Indicates the maximum permissible normal vector deviation angle; if This indicates that the normal vector deviation is too large. This is identified as a noise point feature. Calculate curvature weights : ； in: This represents the actual curvature value at the i-th point. Indicates the maximum permissible curvature threshold; if This indicates that the curvature is too large. This was identified as a feature of interference points. The final weight values ​​are determined by the consistency of the point cloud's normal vectors and its curvature: ； in, This represents the final weight of the i-th point; , Let the normal vector weight coefficient and the curvature weight coefficient be respectively, satisfying the following conditions: ; This represents the consistency weight of the normal vector at the i-th point; This represents the curvature weight at the i-th point; Calculate the ratio of the number of high-weight points within each ROI to the total number of points in the ROI to obtain the percentage of effective points; Set a threshold for the percentage of valid points, retain ROIs with a percentage of valid points greater than or equal to the threshold as qualified ROIs, and remove ROIs with a percentage of valid points lower than the threshold.

8. The method for detecting the warpage of a packaged chip substrate based on 3D vision according to claim 1, characterized in that, Bayesian M-estimation is used to adaptively suppress point cloud noise. Robust principal component analysis is then used to separate the low-rank and sparse components of the point cloud, resulting in a cleaner planar point set. RANSAC is then used to perform planar fitting on the processed point cloud data to generate a reference surface. The specific steps are as follows: Bayesian M-estimation is used to process point cloud data within the qualified ROI. The estimation parameters are adaptively adjusted according to the statistical characteristics of the point cloud data to suppress point cloud noise caused by uneven reflection on the chip surface. Based on the Gaussian distribution assumption of point clouds, the parameters are estimated. Maximize the posterior probability: ； Where the likelihood function Use a robust loss function: ； Where X represents point cloud data; The parameter to be estimated; The parameter is a prior distribution; Let be the coordinate vector of the i-th point cloud data point; For parameter-based The coordinate vector of the predicted point; A robust loss function; The standard deviation of the neighborhood distance of a point; Robust principal component analysis was performed on the denoised point cloud data to decompose the point cloud data into low-rank components and sparse components. The low-rank components correspond to the real substrate plane, and the sparse components correspond to the remaining noise and outliers. The L1 norm was used to separate the low-rank components and sparse components to obtain a purer planar point set. Robust principal component analysis specifically includes: Construct a data matrix from the filtered point cloud data within the ROI; ； in Let i be the three-dimensional coordinates of the world coordinate system of the i-th point; Decompose the data matrix X into a low-rank component L and a sparse component S, and the optimization objective is: ； in The nuclear norm of matrix L is used to constrain the low-rank property of L; Let L1 norm represent the matrix S, which is used to constrain the sparsity of S; For balance parameters; This represents a low-rank component matrix, where each column corresponds to the coordinates of an ideal point on the substrate plane. This represents the sparse component matrix, where each column corresponds to the deviation of the outlier from the ideal plane; After decomposition, the column vector of the low-rank component L is the pure substrate surface point cloud after removing outliers. The RANSAC method is used to fit the plane to obtain the reference plane. A robust loss function is introduced to reduce the impact of outliers on the fitting results. The reference plane is iteratively optimized through a multi-scale refinement strategy. Specifically, at the first scale, RANSAC+Tukey loss is used to perform a global coarse fitting to obtain the fitting plane P1. Then, at the second scale, the interior points of P1 are locally refined to obtain the plane P2. The first scale is larger than the second scale. Through multiple iterations, if the angle between the normal vectors of two adjacent planes is less than a threshold, the iteration is terminated.

9. The method for detecting the warpage of a packaged chip substrate based on 3D vision according to claim 1, characterized in that, Calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference plane, and combine this with a set threshold to quantify the warpage, thereby determining whether the chip exhibits warpage deformation. Specifically: The equation of the reference plane is obtained based on the plane fitting results, and the vertical distance from each effective point in the substrate surface to the reference plane is calculated. ； in, This represents the vertical distance from the i-th valid point on the substrate surface to the reference surface; These are the parameters of the datum surface equation; This represents the three-dimensional coordinates of the i-th point in the world coordinate system. The maximum and minimum values ​​among all vertical distances are counted, and the difference between the maximum and minimum values ​​is used as the quantified value of the warpage of the chip substrate surface. ； in, This represents the maximum vertical distance from all valid points to the reference plane. W is the minimum vertical distance from all valid points to the reference plane; W is the quantified value of warpage. A warpage threshold is set. If the quantized warpage value is less than or equal to the threshold, the chip is determined to have no obvious warpage deformation; if the quantized warpage value is greater than the threshold, the chip is determined to have warpage deformation.

10. A 3D vision-based system for detecting the warpage of a packaged chip substrate, characterized in that, The system for implementing the method according to any one of claims 1 to 9 comprises: The 3D view acquisition module is used to acquire chip depth maps and reference templates, and acquires the chip's true 3D view based on structured light projection technology using optical triangulation. The chip region localization module is used to extract the edges between the reference template and the image to be detected through Canny edge detection, extract the geometric contours by combining Hough transform, locate the complete chip region using template matching algorithm, and remove erroneous contour points by RANSAC verification. The data format conversion module is used to convert depth maps into 3D point cloud data using inverse projection transformation; The point cloud filtering and denoising module is used to filter 3D point cloud data, remove noise points, and retain effective information of the substrate surface. The ROI setting and filtering module is used to set multiple ROIs on the substrate surface, remove residual interference points in the ROIs through dynamic weight sampling, calculate the proportion of effective points, and filter qualified ROIs according to the threshold. The plane fitting module is used to perform plane fitting on the filtered point cloud data. It adopts Bayesian M estimation to adaptively suppress point cloud noise and combines robust principal component analysis to separate the low-rank components and sparse components of the point cloud to obtain a purer plane point set. The processed point cloud data is then fitted with RANSAC to generate a reference surface. The warpage quantification and determination module is used to calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference surface, and combine the set threshold to quantify the warpage and determine whether the chip has bending deformation. The results output module is used to output the test results, including chip warpage parameters and deformation state judgment conclusions.

Images