Packaging chip substrate surface warping degree detection method and system based on 3D vision

Abstract

The invention discloses a packaging chip substrate surface warping degree detection method and system based on 3D vision. The method comprises the following steps: obtaining a real three-dimensional view of a chip through a structured light projection technology in optical triangulation; based on a template matching algorithm, extracting a complete chip area in the depth map; converting the depth map into three-dimensional point cloud data by adopting inverse projection transformation; filtering the obtained point cloud data to remove noise points which do not belong to the substrate surface; setting a plurality of regions of interest on the surface of the substrate, and screening and reserving ROIs (Region of Interest) meeting conditions; performing plane fitting on the reserved ROI by adopting an RANSAC random sampling consensus algorithm to generate a reference plane; and calculating the maximum vertical distance and the minimum vertical distance from each point on the substrate surface to the reference surface, and carrying out quantitative judgment on the warping degree in combination with a set threshold value so as to determine whether the chip has warping deformation or not. According to the method, the warping degree parameter of the single chip can be effectively extracted, and the automatic judgment of the warping state of the chip is realized.

Description

Technical Field

[0001] This invention relates to the field of chip inspection technology, and in particular to a method and system for detecting the warpage of a packaged chip substrate based on 3D vision. Background Technology

[0002] In the post-Moore's Law era, the development of packaging technologies such as 2.5D / 3D integration and advanced packaging has driven chip substrates towards "high density, thinness, and multi-material stacking." During the packaging process, substrates are prone to slight warping due to factors such as mismatched thermal expansion coefficients of materials and residual stress from high-temperature processes. Such warping (even at the micrometer level) can directly lead to defects such as chip stacking misalignment and solder joint failure. Therefore, it is necessary to perform "high-precision, full-field, and non-destructive" warping detection on the substrate surface, as traditional low-precision, localized detection methods are no longer suitable.

[0003] Meanwhile, electronic devices are developing towards miniaturization and high performance, chip packaging density is constantly increasing, the difficulty of matching the thermal expansion coefficient of the substrate is increasing, and the warping problem is becoming more and more prominent. This makes it difficult for traditional low-precision detection methods to meet the needs, and high-precision warping detection technology is urgently needed.

[0004] In the field of chip substrate warpage detection in advanced packaging technology, existing technical solutions still have several shortcomings in complex scenarios: the statistical distance or density filtering methods used in existing solutions have limited adaptability to mixed noise and are difficult to filter out different types of noise; secondly, traditional methods that rely on least squares fitting of the plane are easily affected by outliers, and the fitting accuracy and stability are difficult to guarantee in scenarios with insufficient point cloud purity; the denoising method that relies on multi-scale grid interpolation has a weak ability to suppress residual interference points and reflective noise, and it is difficult to effectively distinguish between the real plane and outliers. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for detecting the warpage of a packaged chip substrate based on 3D vision, which improves the accuracy, efficiency and stability of chip substrate warpage detection and effectively solves the detection error problem caused by interference structures.

[0006] The technical solution to achieve the objective of this invention is: a method for detecting the warpage of a packaged chip substrate based on 3D vision, comprising:

[0007] Obtain chip depth map and reference template, and acquire real three-dimensional view of chip based on structured light projection technology of optical triangulation;

[0008] The edges between the template and the image to be detected are extracted using Canny edge detection, and geometric contours are extracted using Hough transform. The complete chip region is located using a template matching algorithm, and erroneous contour points are removed after RANSAC verification.

[0009] Inverse projection transformation is used to convert the depth map into 3D point cloud data. Based on the camera intrinsic and extrinsic parameter matrices, the depth map pixel coordinates are converted into 3D coordinates in the world coordinate system.

[0010] The 3D point cloud data is filtered to remove interfering structures such as solder balls and pins, while retaining the effective information of the substrate surface.

[0011] Multiple regions of interest (ROIs) are set on the substrate surface. Interference points remaining in the ROIs are removed by dynamic weight sampling. The percentage of valid points is calculated and qualified ROIs are selected according to the threshold.

[0012] Plane fitting was performed on the filtered point cloud data. Bayesian M estimation was used to adaptively suppress point cloud noise. Robust principal component analysis was combined to separate the low-rank and sparse components of the point cloud to obtain a purer planar point set. RANSAC was used to perform plane fitting on the processed point cloud data to generate a reference surface.

[0013] Calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference surface, and combine this with a set threshold to quantify the warpage and determine whether the chip is bent or deformed.

[0014] Furthermore, the structured light projection technology based on optical triangulation to acquire a true three-dimensional view of the chip specifically involves: using a 3D vision system to emit structured light of a specific pattern; the structured light is reflected after being projected onto the surface of the chip substrate; the reflected light signal is captured by a sensor; and the three-dimensional coordinates of each point on the chip surface are calculated based on the principle of optical triangulation to form a true three-dimensional view.

[0015] Furthermore, the steps involve extracting the edges between the template and the image to be detected using Canny edge detection, extracting the geometric contour using Hough transform, locating the complete chip region using a template matching algorithm, and removing incorrectly matched contour points using RANSAC verification. The specific steps are as follows:

[0016] The depth maps of the reference template and the chip under test are converted to grayscale. The Canny edge detection algorithm is used to extract the chip edges. Appropriate high and low thresholds are set to retain clear edge information. Specifically, a high threshold H is set. threshold Low threshold L threshold To preserve the chip's bezel outline;

[0017] The extracted edge map is subjected to Hough transform to identify the rectangular boundary geometric contour of the chip and obtain the coordinate information of the contour points.

[0018] Calculate the shape context descriptor of the contour points, and use the shape context descriptor to measure the local feature similarity between the template contour and the contour of the image to be detected;

[0019] Template matching is performed based on local feature similarity to initially locate the chip region;

[0020] The RANSAC algorithm is used to verify the matching results, remove the contour points of incorrect matches, and select the optimal matching area to achieve accurate positioning of the complete chip area.

[0021] Furthermore, the method of using inverse projection transformation to convert the depth map into 3D point cloud data specifically involves: based on the camera intrinsic and extrinsic parameter matrices, combining the two-dimensional coordinates of each pixel in the depth map with its corresponding depth value, and calculating the 3D coordinates of the pixel in the world coordinate system using the inverse projection transformation formula. The set of all 3D coordinate points constitutes the 3D point cloud data.

[0022]

[0023] in, Represents the two-dimensional coordinates of a pixel in the depth map; This represents the depth value corresponding to a pixel; Indicates the coordinates of the camera's principal point (internal parameter); This indicates the camera's focal length (internal parameter) in the x and y directions. This represents the camera rotation matrix (external parameter). This represents the camera translation vector (extrinsic parameter); This represents the coordinates of a three-dimensional point in the world coordinate system.

[0024] Furthermore, the filtering process for the three-dimensional point cloud data to remove interfering structures such as solder balls and pins, while retaining the effective information of the substrate surface, specifically involves using a combination of statistical filtering and radius filtering. First, statistical filtering removes isolated noise points that deviate from the average distance of the point cloud by more than a set threshold. Then, radius filtering removes the point cloud data corresponding to small-sized interfering structures such as solder balls and pins, while retaining the effective point cloud of the main area of ​​the substrate surface.

[0025] Statistical filtering: Take 50-100 neighboring points for each point, calculate the average distance between the neighboring points, and remove isolated noise points that deviate from the average distance by more than 2-3 times the standard deviation.

[0026] Radius filtering: Set the search radius R search Point clouds with fewer than 8-12 neighboring points (corresponding to interfering structures such as solder balls and pins) are removed, and the effective point clouds on the substrate surface are retained.

[0027] The statistical filtering operation is as follows:

[0028] Calculate a single point Average neighborhood distance:

[0029]

[0030] Calculate the standard deviation of the neighborhood distances of all points:

[0031]

[0032] Noise point judgment criteria (if met, it is removed):

[0033]

[0034] Where: k is the number of neighboring points (value 50-100); For point The j-th neighboring point; The distance is Euclidean; N is the total number of points in the point cloud; This represents the mean of the neighborhood average distances; Indicates a multiple of the standard deviation;

[0035] The radius filtering operation is as follows:

[0036] For point If the number of points in its neighborhood meets the following conditions, it will be removed (corresponding to interference such as solder balls and pins):

[0037]

[0038] Where: r is the search radius; m is the minimum number of neighboring points, with a value of 8-12; This indicates the number of neighboring points that meet the statistical conditions.

[0039] Furthermore, the step of setting multiple ROIs on the substrate surface, removing residual interference points in the ROIs through dynamic weighted sampling, calculating the proportion of valid points, and filtering qualified ROIs according to a threshold is specifically as follows:

[0040] Based on the size and shape of the chip substrate, multiple rectangular ROIs are uniformly set on the substrate surface to ensure that the ROIs cover the clean areas of the substrate surface that do not contain interference information such as pins and solder balls;

[0041] Dynamic weight sampling is performed on the point cloud data within each ROI. Points within the substrate plane are assigned high weights, while interference points such as residual burrs, solder balls, and leads are assigned low weights. Low-weight interference points are then removed to obtain high-purity points within the substrate plane. The dynamic weight calculation is as follows:

[0042] Calculate the consistency weight of the normal vector:

[0043] Calculate the average normal vector of all points within the ROI:

[0044]

[0045] Where: M is the total number of point clouds within the ROI; Let be the unit normal vector of the j-th point, satisfying ;

[0046] Based on the dot product to measure directional similarity, calculate the consistency weight of the normal vector:

[0047]

[0048] in This represents the dot product of the average normal vector and the normal vector of the i-th point, with a range of values. The closer it is to 1, the more consistent the directions are; Indicates the maximum permissible normal vector deviation angle; if This indicates that the normal vector deviation is too large. This was identified as a feature of interference points.

[0049] Calculate curvature weights:

[0050]

[0051] in: This represents the actual curvature value at the i-th point. Indicates the maximum permissible curvature threshold; if This indicates that the curvature is too large. This was identified as a feature of interference points.

[0052] The final weight values ​​are determined by the consistency of the point cloud's normal vectors and its curvature:

[0053]

[0054] in: This represents the final weight of the i-th point; , The normal vector weight coefficients and curvature weight coefficients represent the following: ; This represents the consistency weight of the normal vector at the i-th point; This represents the curvature weight at the i-th point.

[0055] Calculate the ratio of the number of high-weight points within each ROI to the total number of points in the ROI to obtain the percentage of effective points;

[0056] Set a threshold for the percentage of valid points, retain ROIs with a percentage of valid points greater than or equal to the threshold as qualified ROIs, and remove ROIs with a percentage of valid points lower than the threshold;

[0057] Specifically, the ROI setting and screening adopts a partitioning strategy for large-size substrates: 1) Divide the substrate into 16-25 rectangular ROIs evenly according to the substrate size to ensure coverage of the four corners and the central area of ​​the substrate; 2) Set the effective point ratio threshold to ≥80%, and after screening qualified ROIs, fit a local plane for each qualified ROI separately, and then obtain the overall reference plane of the substrate through global splicing to avoid edge fitting errors of large-size substrates.

[0058] Furthermore, Bayesian M-estimation (BME) is used to adaptively suppress point cloud noise, and robust principal component analysis (RPCA) is used to separate the low-rank and sparse components of the point cloud to obtain a purer planar point set. The processed point cloud data is then fitted with RANSAC to generate a reference surface. The specific steps are as follows:

[0059] Bayesian M-estimation is used to process point cloud data within the qualified ROI. The estimation parameters are adaptively adjusted according to the statistical characteristics of the point cloud data to suppress point cloud noise caused by factors such as uneven reflection on the chip surface.

[0060] Based on the Gaussian distribution assumption of point clouds, the parameters are estimated. Maximize the posterior probability:

[0061]

[0062] Where the likelihood function A robust loss function is used (to suppress reflection unevenness noise):

[0063]

[0064] Where: X represents point cloud data; For parameters to be estimated (such as plane model parameters); The parameter prior distribution is used (Gaussian prior is adopted); Let be the coordinate vector of the i-th point cloud data point; For parameter-based The coordinate vector of the predicted point; A robust loss function is used (emphasizing Huber loss to balance Gaussian distribution and outlier suppression). denoted as the standard deviation of the neighborhood distance of a point.

[0065] Robust principal component analysis (RPCA) is performed on the denoised point cloud data to decompose the point cloud data into low-rank components and sparse components. The low-rank components correspond to the real substrate plane, and the sparse components correspond to the remaining noise and outliers. The L1 norm is used to separate the low-rank components and sparse components to obtain a purer planar point set.

[0066] Robust principal component analysis specifically includes:

[0067] Construct a data matrix from the filtered point cloud data (N 3D points) within the ROI;

[0068]

[0069] in Let i be the three-dimensional coordinates (column vector form) of the world coordinate system of the i-th point.

[0070] The data matrix X is decomposed into a low-rank component L (corresponding to the actual substrate plane) and a sparse component S (corresponding to outliers / residual interference). The optimization objective is:

[0071]

[0072] in The nuclear norm (sum of singular values) of matrix L is used to constrain the low rank of L (the three-dimensional data of a planar point cloud can be characterized by 1-2 principal components, and the minimum nuclear norm is equivalent to the minimum rank).

[0073] Let L1 be the L1 norm of matrix S (the sum of the absolute values ​​of all elements), used to constrain the sparsity of S (outliers are few and most elements are 0).

[0074] Indicates the balance parameter (adaptive point cloud quantity);

[0075] This represents the low-rank component matrix (each column corresponds to the coordinates of an ideal point on the substrate surface).

[0076] This represents the sparse component matrix (each column corresponds to the deviation of the outlier from the ideal plane).

[0077] After decomposition, the column vector of the low-rank component L is the pure substrate surface point cloud after removing outliers.

[0078] The RANSAC method is used to fit the plane to obtain the reference plane. A robust loss function is introduced to reduce the impact of outliers on the fitting results. The reference plane is iteratively optimized through a multi-scale refinement strategy. Specifically, at a large scale, a global coarse fit is performed using RANSAC+Tukey loss to obtain the fitted plane P1. Then, at a small scale, a local fine fit is performed on the interior points of P1 to obtain the plane P2. Through multiple iterations, if the angle between the normal vectors of two adjacent planes is less than a threshold, the iteration is terminated, which ultimately improves the accuracy of the plane fitting.

[0079] Furthermore, the maximum and minimum vertical distances from each point on the substrate surface to the reference surface are calculated, and the warpage is quantified by setting a threshold to determine whether the chip has bending deformation. Specifically:

[0080] The equation of the reference plane is obtained based on the plane fitting results, and the vertical distance from each effective point in the substrate surface to the reference plane is calculated.

[0081]

[0082] in: Represents the vertical distance from the i-th effective point on the substrate surface to the reference surface, in μm; The parameters of the datum plane equation are: ; This represents the three-dimensional coordinates of the i-th point in the world coordinate system.

[0083] The maximum and minimum values ​​among all vertical distances are counted, and the difference between the maximum and minimum values ​​is used as the quantified value of the warpage of the chip substrate surface.

[0084]

[0085] in: This represents the maximum vertical distance from all valid points to the reference plane; represents the minimum vertical distance from all valid points to the reference plane; W represents the quantified value of warpage, in μm.

[0086] A warpage threshold is set. If the quantized warpage value is less than or equal to the threshold, the chip is determined to have no obvious warpage deformation; if the quantized warpage value is greater than the threshold, the chip is determined to have warpage deformation.

[0087] A 3D vision-based system for detecting the warpage of a packaged chip substrate, used to implement the above method, the system comprising:

[0088] The 3D view acquisition module is used to acquire chip depth maps and reference templates, and acquires the chip's true 3D view based on structured light projection technology using optical triangulation.

[0089] The chip region localization module is used to extract the edges between the reference template and the image to be detected through Canny edge detection, extract the geometric contours by combining Hough transform, locate the complete chip region using template matching algorithm, and remove erroneous contour points by RANSAC verification.

[0090] The data format conversion module is used to convert depth maps into 3D point cloud data using inverse projection transformation;

[0091] The point cloud filtering and denoising module is used to filter 3D point cloud data, remove noise points, and retain effective information of the substrate surface.

[0092] The ROI setting and filtering module is used to set multiple ROIs on the substrate surface, remove residual interference points in the ROIs through dynamic weight sampling, calculate the proportion of effective points, and filter qualified ROIs according to the threshold.

[0093] The plane fitting module is used to perform plane fitting on the filtered point cloud data. It adopts Bayesian M estimation to adaptively suppress point cloud noise and combines robust principal component analysis to separate the low-rank components and sparse components of the point cloud to obtain a purer plane point set. The processed point cloud data is then fitted with RANSAC to generate a reference surface.

[0094] The warpage quantification and determination module is used to calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference surface, and combine the set threshold to quantify the warpage and determine whether the chip has bending deformation.

[0095] The results output module is used to output the test results, including chip warpage parameters and deformation state judgment conclusions.

[0096] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention integrates the chip region localization scheme of Canny edge detection, Hough transform, shape context descriptor and RANSAC verification, which greatly improves the accuracy of chip region localization and effectively avoids the positioning error caused by edge blurring and interference structure; (2) The combination of statistical filtering and radius filtering is used to denoise the chip region, and the multi-level noise suppression strategy of dynamic weight sampling and Bayesian M estimation (BME) and robust principal component analysis (RPCA) can efficiently remove various interference factors such as solder balls, pins and uneven reflection, and obtain high purity substrate surface point cloud data; (3) The plane fitting method of RANSAC + multi-scale refinement is used to improve the fitting accuracy of the reference plane. Combined with the warp measurement method of the maximum and minimum vertical distance difference, the high-precision warp detection at the micron level is realized, which meets the detection requirements of the packaging field; (4) The entire detection process is highly automated and efficient. It is a non-contact detection method that will not damage the chip and is suitable for industrial online detection scenarios. Attached Figure Description

[0097] Figure 1 This is an overall flowchart of the 3D vision-based method for detecting the warpage of a packaged chip substrate according to the present invention.

[0098] Figure 2 This is a schematic diagram of the chip area positioning process;

[0099] Figure 3This is a schematic diagram of the point cloud filtering and ROI selection process;

[0100] Figure 4 This is a schematic diagram of the plane fitting process;

[0101] Figure 5 A visualization of the chip's original point cloud;

[0102] Figure 6 A visualization of the chip's original point cloud (showing noise data);

[0103] Figure 7 A visualization of the substrate surface after noise removal from the chip point cloud data;

[0104] Figure 8 The image shows the point cloud of the chip after fitting the plane (the red part is the reference plane). Detailed Implementation

[0105] To address the shortcomings of existing technologies, this invention employs multi-stage technological innovations, utilizing an algorithm combining statistical filtering and radius filtering to more flexibly handle different types of noise interference. A fitting method based on ROI clean point clouds and RANSAC random sampling reduces the impact of outliers, while multi-level optimization further improves the accuracy of the baseline fitting. A combined scheme of Dynamic Weighted Sampling (DWS), Bayesian M-estimation (BME), and RPCA efficiently removes residual interference points and suppresses reflective noise, while accurately separating the real plane from outliers. The multi-scale refinement optimization logic, through large-scale initial fitting and small-scale iterative optimization, achieves a gradual improvement in fitting accuracy. The overall solution is more suitable for advanced packaging chip substrate warpage detection scenarios characterized by "high interference, high accuracy requirements, and strong online detection needs," meeting the core requirements of packaging processes for "high precision, high efficiency, and non-destructive testing," thus becoming one of the effective methods for substrate warpage detection. The method of this invention can effectively extract the warpage parameters of a single chip, realize the automatic determination of the chip warpage state, and has high-precision and high-efficiency detection performance. It is suitable for industrial scenarios such as semiconductor packaged chip inspection, precision electronic component assembly quality assessment, precision component surface inspection, and high-precision PCB manufacturing process monitoring.

[0106] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. A detailed description of the 3D vision-based method and system for detecting the warpage of a packaged chip substrate is provided.

[0107] The 3D vision-based packaging chip substrate warpage detection system of this embodiment includes a three-dimensional view acquisition module, a chip region positioning module, a data format conversion module, a point cloud filtering and denoising module, a ROI setting and filtering module, a plane fitting module, a warpage quantification and judgment module, and a result output module. Specifically: the three-dimensional view acquisition module uses a 3D vision sensor based on optical triangulation, emitting structured light; the camera intrinsic parameter matrix is ​​determined through calibration experiments, and the extrinsic parameter matrix is ​​obtained through calibration of the relative position of the camera and the chip; the chip region positioning module, data format conversion module, point cloud filtering and denoising module, ROI setting and filtering module, plane fitting module, and warpage quantification and judgment module are implemented through software programming and run on an industrial control computer; the result output module includes a display and a data storage unit for real-time display of detection results and storage of detection data.

[0108] Combination Figures 1-4 The 3D vision-based method for detecting the warpage of a packaged chip substrate in this embodiment includes the following steps:

[0109] Step 1: Obtain the chip depth map and reference template. Using structured light projection technology based on optical triangulation, acquire a true 3D view of the chip. Fix the packaged chip to be inspected on the inspection platform, ensuring the chip substrate surface is flat and unobstructed. Activate the 3D view acquisition module. The 3D vision sensor emits structured light onto the chip substrate surface. The structured light is reflected by the substrate surface and captured by the sensor. Based on the principle of optical triangulation, calculate the 3D coordinates of each point on the chip surface according to the angle difference between the emitted and reflected light, forming a true 3D view of the chip. Simultaneously, import the reference template for this chip model. The reference template is a chip depth map without warping defects.

[0110] Step 2: Extract the edges between the template and the image to be detected using Canny edge detection, combine this with Hough transform to extract the geometric contours, and use a template matching algorithm to locate the complete chip region. Verify with RANSAC to remove incorrectly matched contour points. Perform grayscale processing on the depth maps of the reference template and the chip to be detected, and use the Canny edge detection algorithm to extract edges, setting a high threshold H. threshold Low threshold L thresholdThe process preserves clear chip edge information; Hough transform is applied to the extracted edges, and the accumulator threshold in the Hough space is set to 100 to identify the rectangular geometric contour of the chip and obtain the coordinate data of the contour points; the shape context descriptor of each contour point is calculated, and multiple angle and distance intervals are divided in polar coordinates with the contour point as the center, and the number of contour points in each interval is counted to form a shape context descriptor; the distance between the shape context descriptors of the template contour and the contour of the image to be detected is calculated to measure the local feature similarity, and template matching is performed based on the local feature similarity to initially locate the chip region; the RANSAC algorithm is used to verify the matching results, such as setting the interior point threshold to 2 pixels, removing incorrectly matched contour points, and selecting the optimal matching region to achieve accurate positioning of the complete chip region, with the positioning error controlled within ±1 pixel.

[0111] Step 3: Convert the depth map into 3D point cloud data using inverse projection transformation. Based on the intrinsic parameter matrix (focal length, principal point coordinates, etc.) and extrinsic parameter matrix (rotation matrix, translation vector, etc.) of the 3D vision sensor, combine the (u, v) 2D coordinates of each pixel in the depth map with the corresponding depth value z to obtain the chip's point cloud map, such as... Figure 5 As shown.

[0112] Step 4: The initial 3D point cloud map carries a large amount of noisy data, such as solder balls, pins, etc. Figure 6 As shown, the denoising process mainly involves filtering the 3D point cloud data to remove interfering structures such as solder balls and pins, while retaining the effective information of the substrate surface. A statistical filtering algorithm is used for initial denoising of the 3D point cloud data, such as setting the number of neighboring points to 50 and the standard deviation to 2.0, to remove isolated noise points whose distance from the average distance of neighboring points exceeds 2 times the standard deviation. A radius filtering algorithm is then used to further remove interfering structures, with the search radius set to R. search The minimum number of neighboring points is 10. Point cloud data corresponding to small-sized interfering structures such as solder balls and leads are removed, and the effective point cloud of the main area of ​​the substrate surface is retained. The processed point cloud map is shown below. Figure 7 As shown.

[0113] Step 5: Set multiple ROIs on the substrate surface. Use dynamic weight sampling to remove residual interference points in the ROIs, calculate the effective point ratio, and filter qualified ROIs according to the threshold. Based on the size of the chip substrate surface, set N ROIs evenly on the substrate surface to ensure coverage of the entire substrate surface. Perform dynamic weight sampling on the point cloud data in each ROI. Based on the normal vector and curvature characteristics of the point cloud, assign high weight (weight value of 1.0) to points in the substrate surface (with consistent normal vectors and small curvature), and assign low weight (e.g., weight value of 0.1) to residual burrs, solder balls, pins, and other interference points (with disordered normal vectors and large curvature). Remove interference points with weight values ​​lower than 0.5 to obtain high purity points in the substrate surface. Calculate the ratio of the number of high-weight points in each ROI to the total number of points in the ROI to obtain the effective point ratio. Set an effective point ratio threshold and retain ROIs with an effective point ratio greater than the threshold as qualified ROIs. In this embodiment, all ROIs are qualified ROIs.

[0114] Step 6: Adaptive suppression of point cloud noise is achieved using Bayesian M-estimation (BME). Robust Principal Component Analysis (RPCA) is used to separate the low-rank and sparse components of the point cloud. BME is applied to the point cloud data within the qualified ROI. Based on the Gaussian distribution characteristics of the point cloud data, the estimation parameters are adaptively adjusted to suppress point cloud noise caused by uneven reflection on the chip surface. Robust Principal Component Analysis (RPCA) is then performed on the denoised point cloud data to construct the covariance matrix. Principal components are obtained through eigenvalue decomposition, decomposing the point cloud data into low-rank components (corresponding to the actual substrate plane) and sparse components (corresponding to residual noise and outliers). L1 grading is then applied. Norm separation separates low-rank and sparse components, yielding a pure planar point set. RANSAC is used for planar fitting to obtain a reference surface. A robust loss function is introduced to reduce the impact of outliers on the fitting results. A multi-scale refinement strategy iteratively optimizes the reference surface: first, RANSAC+Tukey loss is used for global coarse fitting at a large scale, then local fine fitting is performed at a small scale. Through multiple iterations, the accuracy of the planar fitting is ultimately improved, resulting in the fitted reference surface as shown below. Figure 8 As shown in the red area.

[0115] Step 7: Calculate the maximum and minimum vertical distances from each point on the substrate surface to the reference plane. Combine this with a set threshold to quantify the warpage and determine if the chip has bending deformation. Obtain the equation of the reference plane based on the plane fitting results. Calculate the vertical distance from each effective point on the substrate surface to the reference plane. Statistically find the maximum and minimum values ​​among all vertical distances to calculate the quantified warpage value. Based on the packaging requirements of this chip model, set the warpage threshold to W. threshold .

[0116] Step 8: Display the test results in real time through the display of the result output module, including chip warpage parameters and deformation state judgment conclusions; at the same time, store the test data (including 3D point cloud data, reference surface parameters, warpage values, judgment results, etc.) to the data storage unit for subsequent traceability and analysis.

Claims

1.A 3D vision-based method for detecting warpage of a package substrate surface, characterized in that, The method comprises the following steps: Obtain the chip depth map and the reference template, and collect the real three-dimensional view of the chip based on the structured light projection technology of optical triangulation; Extract the edges of the reference template and the to-be-detected image through Canny edge detection, extract the geometric contour by combining Hough transformation, locate the complete chip region by using a template matching algorithm, and remove the false matching contour points through RANSAC verification; Convert the depth map into three-dimensional point cloud data by using inverse projection transformation, convert the depth map coordinates into three-dimensional coordinates in the world coordinate system based on the camera intrinsic matrix and the extrinsic matrix; Filter the three-dimensional point cloud data, remove the noise points, and retain the effective information of the substrate surface; Set multiple regions of interest on the substrate surface, remove the residual interference points in the ROI by dynamic weight sampling, calculate the proportion of effective points, and screen the qualified ROIs according to a threshold value; Perform plane fitting on the screened point cloud data, adaptively suppress the point cloud noise by using Bayesian M estimation, separate the low-rank component and the sparse component of the point cloud by combining robust principal component analysis, obtain a purer plane point set, and perform plane fitting on the processed point cloud data by using RANSAC to generate the reference surface; Calculate the maximum vertical distance and the minimum vertical distance from each point on the substrate surface to the reference surface, quantify the warping degree by combining a set threshold value, and determine whether the chip has warping deformation. 2.The 3D vision-based package substrate warpage detection method of claim 1, wherein, Collect the real three-dimensional view of the chip based on the structured light projection technology of optical triangulation, specifically as follows: emit structured light by using a 3D vision system, the structured light is reflected after being projected onto the chip substrate surface, the reflected light signal is captured by a sensor, the three-dimensional coordinates of each point on the chip surface are calculated based on the principle of optical triangulation, and a real three-dimensional view is formed. 3.The 3D vision-based package substrate warpage detection method of claim 1, wherein, Extract the edges of the reference template and the to-be-detected image through Canny edge detection, extract the geometric contour by combining Hough transformation, locate the complete chip region by using a template matching algorithm, and remove the false matching contour points through RANSAC verification, specifically as follows: Perform grayscale processing on the depth map of the reference template and the to-be-detected chip, extract the chip edge by using the Canny edge detection algorithm, and set high and low threshold values to retain clear edge information; Perform Hough transformation on the extracted edge map, identify the rectangular frame geometric contour of the chip, and obtain the coordinate information of the contour points; Calculate the shape context descriptor of the contour points, measure the local feature similarity between the template contour and the to-be-detected image contour by using the shape context descriptor; Perform template matching based on the local feature similarity, and preliminarily locate the chip region; Verify the matching result by using the RANSAC algorithm, remove the false matching contour points, screen the optimal matching region, and realize accurate positioning of the complete chip region. 4.The 3D vision-based package substrate warpage detection method of claim 1, wherein, Convert the depth map into three-dimensional point cloud data by using inverse projection transformation, specifically as follows: combine the two-dimensional coordinates of each pixel point in the depth map and the corresponding depth value, calculate the three-dimensional coordinates of the pixel point in the world coordinate system by using the inverse projection transformation formula, and all three-dimensional coordinate points constitute three-dimensional point cloud data; ； wherein, represents the two-dimensional coordinates of a pixel in the depth map; represents the depth value corresponding to the pixel; represents the principal point coordinates of the camera; represents the focal length of the camera in the x, y directions; represents the rotation matrix of the camera; represents the translation vector of the camera; represents the three-dimensional point coordinates in the world coordinate system. 5.The 3D vision-based package substrate warpage detection method of claim 1, wherein, The three-dimensional point cloud data is filtered to remove noise points and retain effective information of the substrate surface. Specifically, a method combining statistical filtering and radius filtering is adopted. First, statistical filtering is used to remove isolated noise points deviating from the average distance of the point cloud by more than a set threshold. Then, radius filtering is used to remove point cloud data corresponding to interference structures such as solder balls and pins, and to retain effective point clouds in the main area of the substrate surface. 6.The 3D vision-based package substrate warpage detection method of claim 5, wherein, The statistical filtering operation is specifically as follows: Computing the neighborhood average distance of a single point :​ ； calculating the standard deviation of all point neighborhood distances : ； The noise point judgment condition is as follows: if it is met, it is removed: ； wherein: k is the number of neighborhood points; is the jth neighborhood point of ; denotes the Euclidean distance; N is the total number of points of the point cloud; denotes the mean of the neighborhood average distances; denotes the standard deviation multiple; The radius filtering operation is specifically as follows: For a point , it is removed if the number of points in its neighborhood satisfies the following condition: ； Wherein: r is the search radius; m is the minimum number of neighborhood points; Indicates the number of neighborhood points that meet the condition. 7.The 3D vision-based package substrate warpage detection method of claim 1, wherein, A plurality of regions of interest (ROIs) are set on the substrate surface. Interference points remaining in the ROIs are removed by dynamic weight sampling. The proportion of effective points is calculated and filtered according to a threshold to obtain qualified ROIs. Specifically: According to the size and shape of the chip substrate, a plurality of rectangular ROIs are uniformly set on the substrate surface to ensure that the ROIs cover the pure regions of the substrate surface that do not contain interference information such as pins and solder balls. Dynamic weight sampling is performed on the point cloud data in each ROI. High weight is given to the points in the substrate surface, and low weight is given to the remaining noise points. Low-weight noise points are removed to obtain points in the substrate surface. The dynamic weight calculation is specifically as follows: Calculate the average normal vector of all points within the ROI : ； Wherein, M is the total number of point clouds in the ROI; is the unit normal vector of the jth point, satisfying ; Based on the dot product to measure the direction similarity, the normal vector consistency weight is calculated : ； wherein denotes the dot product of the average normal vector and the normal vector of the i-th point, taking values in the range , the closer to 1 the more consistent the directions are; denotes the maximum allowed normal vector deviation angle; if , the normal vector deviation is too large, , the point is determined to be a noise point feature; Computing curvature weights : ； wherein: represents the actual curvature value of the i-th point, represents the maximum allowed curvature threshold; if , the curvature is too large, , the point is determined to be a feature of an interference. The maximum weight value is determined by the consistency of the normal vector of the point cloud and the curvature: ； wherein, denotes the minimum weight of the i-th point; , denote the normal vector weight coefficient and the curvature weight coefficient, respectively, satisfying ; denotes the normal vector consistency weight of the i-th point; denotes the curvature weight of the i-th point; The ratio of the number of high-weight points in each ROI to the total number of points in the ROI is calculated to obtain the proportion of effective points. An effective point proportion threshold is set. ROIs with an effective point proportion greater than or equal to the threshold are retained as qualified ROIs, and ROIs with an effective point proportion less than the threshold are removed. 8.The 3D vision based package substrate warpage detection method of claim 1, wherein, Bayesian M estimation is used to adaptively suppress point cloud noise. Robust principal component analysis is used to separate the low-rank component and the sparse component of the point cloud to obtain a purer plane point set. RANSAC is used to fit the plane to the processed point cloud data to generate a reference surface. The specific steps are as follows: The point cloud data in the qualified ROI is processed using Bayesian M estimation. The estimation parameters are adaptively adjusted according to the statistical characteristics of the point cloud data to suppress point cloud noise caused by uneven chip surface reflection. Estimate parameters based on the assumption of Gaussian distribution of point cloud Maximum a posteriori probability of the parameters: ； where the likelihood function with a robust loss function: ； wherein X is the point cloud data; is the parameter to be estimated; is the parameter prior distribution; is the coordinate vector of the i-th point cloud data point; is the coordinate vector of the predicted point based on the parameter ; is the robust loss function; is the standard deviation of the point neighborhood distances; Robust principal component analysis is performed on the denoised point cloud data to decompose the point cloud data into a low-rank component and a sparse component. The low-rank component corresponds to the real substrate surface plane, and the sparse component corresponds to the remaining noise and outliers. L1 norm is used to separate the low-rank component and the sparse component to obtain a purer plane point set. The robust principal component analysis is specifically as follows: The filtered point cloud data in the ROI is used to construct a data matrix. ； wherein Xi is the three-dimensional coordinate of the world coordinate system for the i-th point; The data matrix X is decomposed into a low-rank component L and a sparse component S. The optimization objective is as follows: ； wherein denotes the nuclear norm of matrix L, used to constrain the low-rankness of L; denotes the LI norm of matrix S, used to constrain the sparsity of S; is a balancing parameter; denotes the low-rank component matrix, each column of which corresponds to the ideal point coordinates on the substrate plane; denotes the sparse component matrix, each column of which corresponds to the deviation of outlier points from the ideal plane; After decomposition, the column vectors of the low-rank component L are the pure substrate surface point clouds after removing outliers. The RANSAC method is used for plane fitting to obtain the reference surface, a robust loss function is introduced to reduce the influence of abnormal points on the fitting result, and a multi-scale refinement strategy is used for iterative optimization of the reference surface. Specifically, first, a global rough fitting is performed at a first scale using RANSAC + Tukey loss to obtain a fitting plane P1, then a local fine fitting is performed at a second scale on the inliers of P1 to obtain a plane P2, the first scale is greater than the second scale, through multiple iterations, if the angle between the normal vectors of the adjacent two planes is less than a threshold, the iteration is terminated. 9.The 3D vision based package substrate warpage detection method of claim 1, wherein, The maximum and minimum vertical distances of each point on the substrate surface to the reference surface are calculated, and a threshold is set to quantify the warpage, to determine whether the chip has warpage deformation. Specifically: According to the plane fitting result, the equation of the reference surface is obtained, and the vertical distance of each effective point in the substrate surface to the reference surface is calculated; ； wherein, represents the vertical distance from the i-th substrate surface effective point to the reference surface; is the reference surface equation parameter; represents the three-dimensional coordinates of the i-th point in the world coordinate system; The maximum and minimum values of all vertical distances are counted, and the difference between the maximum and minimum values is taken as the warpage quantization value of the chip substrate surface; ； wherein, is the maximum value of the vertical distance of all valid points to the reference plane; is the minimum value of the vertical distance of all valid points to the reference plane; W is the warpage quantification value; A warpage threshold is set, if the quantized warpage value is less than or equal to the threshold, it is determined that the chip has no obvious warpage deformation; if the quantized warpage value is greater than the threshold, it is determined that the chip has warpage deformation. 10.A 3D vision-based system for detecting warpage of a package substrate surface, characterized in that, The system is used to implement the method of any one of claims 1-9, and comprises: A three-dimensional view acquisition module is configured to acquire a chip depth map and a reference template, and to acquire a real three-dimensional view of the chip based on a structured light projection technology of optical triangulation; A chip region positioning module is configured to extract edges of the reference template and the to-be-detected image through Canny edge detection, to extract a geometric contour by combining a Hough transform, and to position a complete chip region by using a template matching algorithm and by eliminating false matching contour points through RANSAC verification; A data format conversion module is configured to convert the depth map into three-dimensional point cloud data by using an inverse projection transformation; A point cloud filtering and denoising module is configured to filter and denoise the three-dimensional point cloud data, and to retain effective information of the substrate surface; An ROI setting and screening module is configured to set multiple ROIs on the substrate surface, to eliminate residual interference points in the ROIs by dynamic weight sampling, to calculate a proportion of effective points, and to screen qualified ROIs according to a threshold; A plane fitting module is configured to perform plane fitting on the screened point cloud data, to adaptively suppress point cloud noise by using Bayesian M estimation, to separate low-rank components and sparse components of the point cloud by combining robust principal component analysis, to obtain a purer plane point set, and to generate a reference surface by performing plane fitting on the processed point cloud data by using RANSAC; A warpage quantification and judgment module is configured to calculate the maximum and minimum vertical distances of each point on the substrate surface to the reference surface, to quantify the warpage by combining a set threshold, and to determine whether the chip has warpage deformation; A result output module is configured to output a detection result, including a chip warpage parameter and a deformation state judgment conclusion.

Images