Packaged chip solder ball array quality detection method and system based on perspective transformation

Abstract

This invention discloses a method and system for quality inspection of solder ball arrays in packaged chips based on perspective transformation. The method includes: employing a nine-grid partitioning and layered compensation strategy to achieve image homogenization and contrast enhancement; achieving high-precision extraction of solder ball edges through edge ROI and multi-directional Sobel edge enhancement; introducing chip outer contour shape constraints to perform overall linear fitting of the solder ball array edges; achieving dynamic template mapping based on perspective transformation to correct chip posture distortion; employing a two-level registration strategy to achieve global alignment between the template and the image to be inspected; achieving precise template calibration through a three-level intelligent shielding mechanism and interactive fine-tuning; and completing quantitative calculation and automatic judgment of defects based on a solder ball quality evaluation model. This invention enables high-precision and robust inspection of solder ball arrays in packaged chips such as BGA / CSP in industrial environments, significantly reducing the false negative and false positive rates, and is suitable for industrial online automated inspection.

Description

Technical Field

[0001] This invention relates to the fields of machine vision inspection and semiconductor packaging inspection technology, and in particular to a method and system for quality inspection of packaged chip solder ball arrays based on perspective transformation. Background Technology

[0002] In semiconductor packaging, the quality of the solder ball array directly affects the electrical connection performance and reliability of the chip. Existing solder ball detection methods mostly rely on simple image processing or template matching techniques, which are easily affected by factors such as chip orientation shifts and solder ball shape changes in practical applications, leading to a decrease in detection accuracy.

[0003] Especially in real-world production environments, the variety of chip types, the different trays supporting the chips, the diverse solder ball distributions, and the complex surface textures of the chips make it difficult for traditional edge detection and template matching methods to reliably extract solder ball features. Furthermore, with chips placed on trays, cameras capture images of multiple chips (3×3 or 4×4 at a time, the exact number depending on factors like camera resolution). Chips located at the four corners of the tray are prone to significant geometric distortion, which traditional methods fail to effectively correct, resulting in large solder ball positioning deviations and high rates of missed and false detections. In addition, while solder ball arrays have a regular topological structure, existing methods often fail to fully utilize this structural information for global constraints, thus reducing registration accuracy.

[0004] Therefore, it is necessary to propose a solder ball array detection method that can eliminate shooting distortion through perspective transformation, achieve high-precision registration by combining geometric constraints and topological structure, and at the same time have reliable quality assessment capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for quality inspection of packaged chip solder ball arrays based on perspective transformation, so as to solve the problems of strong light sensitivity, large distortion caused by shooting angle and tray position, low registration accuracy and unstable quality judgment in the prior art.

[0006] The technical solution to achieve the objective of this invention is: a method for inspecting the quality of solder ball arrays in packaged chips based on perspective transformation, comprising:

[0007] (1) Acquire the original image of the chip and perform preprocessing. Use a nine-grid partitioning and layer compensation strategy to achieve image homogenization and contrast enhancement.

[0008] (2) The chip edges of the image under test are roughly extracted by using edge ROI combined with multi-directional Sobel operator to obtain the chip edge point set;

[0009] (3) By introducing chip shape constraints, the edge points are linearly fitted as a whole to complete the fine extraction of the solder ball array edge and obtain the chip boundary line and the corresponding corner coordinates;

[0010] (4) Based on the four corner points of the chip, calculate the perspective transformation matrix and perform perspective transformation on the image to convert the chip image from the original perspective view to the standard frontal view.

[0011] (5) Construct the solder ball topology of the virtual template according to the specification parameters, and construct the virtual template size / boundary according to the image size and chip position, so that the virtual template is initially aligned with the solder ball array in the calibration image;

[0012] (6) Using the topological features of the solder ball array, the virtual template and the candidate solder ball positions in the calibration image are matched. By the correspondence between the center point of the solder ball in the virtual template and the center point of the candidate solder ball in the calibration image, a local region is divided for each solder ball position. Solder ball features are extracted in the local region, and the center position of the solder ball is accurately located by the local feature optimization method.

[0013] (7) Based on precise positioning, point set registration is completed based on centroid constraints and singular value decomposition to achieve global alignment between the virtual template and the measured solder ball array;

[0014] (8) Extract the solder ball contour through three-level intelligent shielding and morphological processing;

[0015] (9) The quality of solder balls is determined based on the gray difference between the inner and outer circles, and the chip solder ball defect detection results and quality assessment data are output.

[0016] Further, in step (1), the image is subjected to a nine-grid partitioning illumination assessment and layered compensation processing. The nine-grid partitioning refers to dividing the image into 3×3 regions, and calculating the mean intensity, standard deviation, and histogram skewness for each region. The mean intensity is used to construct an illumination unevenness index; the standard deviation is used to characterize the degree of brightness fluctuation within the region, judging the intensity of local reflections and noise; and the histogram skewness is used to judge the overall brightness distribution trend of the region, identifying dark and overexposed areas. The illumination unevenness index is constructed based on the mean intensity, and its calculation formula is:

[0017]

[0018] in, This represents the maximum average brightness of each sub-region. This represents the minimum value. The image is represented by the overall average brightness. Subsequently, the image is subjected to layered compensation processing according to the illumination unevenness index. At the global layer, the image is enhanced by the contrast-limited adaptive histogram equalization algorithm, and the contrast cropping parameters are dynamically adjusted according to the illumination unevenness. At the local layer, Gaussian smoothing is performed on overexposed areas, and gamma correction is performed on underexposed areas, thereby achieving overall optimization of the image brightness distribution.

[0019] Furthermore, in step (2), multiple regions of interest are set in the four edge regions of the chip, and edge detection is performed in each region of interest using a polarized Sobel operator, wherein:

[0020] For chip edges in different directions, the corresponding gradient calculation direction and polarity are selected based on the direction of edge grayscale change, thus retaining only the gradient response that conforms to the chip boundary characteristics to suppress non-target edge interference and improve the accuracy of edge extraction. Specifically, using edge ROI and multi-directional Sobel operators for coarse edge extraction involves setting multiple regions of interest (ROIs) in the upper, lower, left, and right edge regions of the chip image and performing edge detection within these regions. Then, a polarized Sobel operator is used to select the corresponding gradient direction and polarity based on the direction of the chip edge, retaining only the gradient response that conforms to the boundary characteristics, thereby obtaining a stable set of edge points.

[0021] Furthermore, in step (3), the overall linear fitting process of the chip edge includes: representing the chip boundary as a model of multiple straight lines, each line being represented in the following form:

[0022]

[0023] in, , , They represent the first Parameters of the boundary lines;

[0024] Then, using the distance from the edge point to the corresponding line as the error metric, an overall optimization objective function is constructed:

[0025]

[0026] in, Indicates the first The coordinates of the edge points Indicates that the edge point belongs to the first... The weighting coefficients of each boundary;

[0027] During the optimization process, shape constraints of the rectangular chip structure are introduced to ensure that opposite sides are parallel and adjacent sides are perpendicular. Under these constraints, the objective function is jointly optimized and solved to obtain the parameters of the four boundary lines that satisfy the geometric characteristics of the rectangle.

[0028] Furthermore, the coordinates of the four corner points of the chip are obtained by calculating the intersection of two adjacent boundary lines. The corner points are obtained by solving the following system of linear equations:

[0029]

[0030] in, and This represents two adjacent boundary lines. , , They represent the boundary lines respectively. The parameters, , , They represent the boundary lines respectively. The parameters; solving the system of equations yields the point where the two lines intersect, which is the corner point, with coordinates of... .

[0031] Furthermore, in step (4), the perspective transformation is achieved by establishing a mapping relationship between the original image coordinates and the corrected image coordinates, which is expressed as:

[0032] ;

[0033] in, Represents points in the original image. Indicates the corrected point. This is the perspective transformation matrix; this transformation maps the image from perspective space to standard planar space, thereby eliminating geometric distortion.

[0034] Furthermore, in step (5), the virtual template generation process is constructed based on the specification parameters of the packaged chip. The specification parameters include the number of rows and columns of the solder ball array and the spacing information between adjacent solder balls, thereby forming a template point set with a regular topology.

[0035] Based on the specified parameters, a solder ball array model is established in a two-dimensional plane coordinate system, wherein the template is... line, number The center coordinates of the solder balls in the column are represented as follows:

[0036]

[0037] in, This indicates the center coordinates of the corresponding solder ball in the template. and These represent the row and column indices of the solder balls in the array, respectively. and These represent the spacing parameters of adjacent solder balls in the horizontal and vertical directions, respectively, and the spacing parameters are derived from the specification parameters;

[0038] The set of template points satisfies the array topology consistency constraint, that is, the distance between any two adjacent template points satisfies:

[0039] ;

[0040] To ensure the regularity and consistency of the template structure in space; after the template point set is constructed, a corresponding local template area is generated based on the template center point for subsequent solder ball positioning and registration processes, so that the virtual template can establish a spatial correspondence with the perspective-corrected chip image.

[0041] Furthermore, in step (6), the precise localization based on local feature optimization and error minimization specifically includes the following process:

[0042] Step 6.1: Generate a set of virtual template solder ball center points based on the virtual template. ,in This indicates the center coordinates of the corresponding solder ball in the template, which are located at the center point of the template. Obtained by one-dimensional mapping, Indicates the number of the solder ball in the array;

[0043] Step 6.2: Extract the set of candidate solder ball center points in the perspective-corrected image based on the topological relationship. These candidate solder ball center points are obtained through grayscale features, connected component analysis, or local extremum detection methods, thus forming the set of candidate solder ball center points in the corrected image. ,in This indicates the center coordinates of the corresponding solder ball in the corrected image. Indicates the number of the solder ball in the array;

[0044] Step 6.3: Correct the center point of the candidate solder ball in the image. A local neighborhood window is established centered on the solder ball. The size of the selected local neighborhood window should be smaller than the standard spacing between solder balls. Simultaneously, the local neighborhood window of each solder ball is processed in a left-to-right, top-to-bottom order to ensure the uniqueness of each solder ball's edge. Within this area, the edge points of the solder ball are extracted using a sub-pixel-level edge detection method, forming the local point set corresponding to that solder ball. ,in Indicates the solder ball number. Indicates the edge point index. The image coordinates of the corresponding edge points;

[0045] Step 6.4: For each solder ball's neighborhood window, fit the solder ball's contour using a circular model, expressed as follows:

[0046] ;

[0047] in, For local point sets Image coordinates of subpixel edge points within the area; Indicates the coordinates of the solder ball center. Indicates the radius of the solder ball;

[0048] Step 6.5: Construct a least-squares error function for the set of edge points:

[0049] ;

[0050] The center position of the solder ball is determined by minimizing the sum of squared residuals from the edge points to the circular profile, thus obtaining the precise center coordinates of each solder ball in the corrected image. This forms a set of precise center coordinates for the solder ball array in the corrected image. ,in Indicates the solder ball number. For the first The coordinates of the center point of each solder ball.

[0051] Furthermore, in step (7), point set registration is completed based on centroid constraints and singular value decomposition to achieve global alignment between the virtual template and the measured solder ball array. The specific process includes:

[0052] Step 7.1: Based on the center points of the virtual template solder balls and the measured center points of the solder balls, solve for the global centroid of these two sets of points respectively:

[0053] ;

[0054] in, For the centroid of the virtual template, Let M be the centroid of the image to be inspected, and M be the total number of solder balls. Indicates the corresponding number in the virtual template The center position of each solder ball Indicates the first in the corrected image The precise center position of each solder ball;

[0055] Step 7.2: Decentralize the two sets of points to eliminate the influence of the translation components:

[0056] ;

[0057] ;

[0058] in, For the corresponding number in the virtual template The center position of each solder ball after decentralization For the corresponding number in the image to be inspected The center position of each solder ball after decentralization For the decentralized virtual template solder ball center point set, This is the set of candidate solder ball center points after decentralization; at this point, the overall translation components of the two sets of points are eliminated, and only the relative position information is retained to avoid translation interference with the solution of the rotation matrix;

[0059] Step 7.3: Construct the covariance matrix based on the decentralized virtual template solder ball center point set and the candidate solder ball center point set. This matrix represents the linear relationship between two sets of points and is the core basis for solving the optimal rigid rotation matrix.

[0060] ;

[0061] in, It is a 2×2 covariance matrix, containing the correlation features of the two sets of points; The transpose matrix of the decentralized virtual template solder ball center point set;

[0062] Step 7.4: Perform singular value decomposition on the covariance matrix, decomposing the matrix into a product of an orthogonal matrix and a diagonal matrix; directly extract the optimal rigid rotation matrix from the decomposition result to ensure that the rotation transformation minimizes the matching error between the two sets of points.

[0063] ;

[0064] in, Represents a left singular orthogonal matrix; Represents a singular value diagonal matrix; Represents a right singular orthogonal matrix; Represents a right singular orthogonal matrix Transpose of;

[0065] Step 7.5: Calculate the optimal rigid rotation matrix using the orthogonal matrix obtained from singular value decomposition. This matrix achieves optimal angular alignment between the virtual template solder ball center point set and the candidate solder ball center point set, while satisfying the orthogonality constraint of the rigid transformation:

[0066] ;

[0067] in, It is the transpose of the left singular orthogonal matrix U;

[0068] Step 7.6: Combining the global centroids of the two point sets with the optimal rigid rotation matrix, solve for the two-dimensional global translation vector; the two-dimensional global translation vector is used to compensate for the global positional deviation of the two point sets, completing the solution for the translation components of the rigid transformation:

[0069] ;

[0070] in, It is a two-dimensional global translation vector, representing the overall positional offset between the virtual template solder ball center point set and the candidate solder ball center point set;

[0071] Step 7.7: Combine the optimal rigid rotation matrix with the two-dimensional global translation vector to form a rigid transformation model, and transform the virtual template point set to achieve global alignment between the template point set and the measured point set, thus completing the initial registration.

[0072] ;

[0073] in, Indicates the aligned first A virtual template solder ball center point;

[0074] Step 7.8: Output the optimal rigid transformation parameters and the globally aligned set of center points of the solder balls in the image to be inspected, and complete the point set registration of the solder ball array, providing a reliable benchmark for subsequent solder ball contour extraction and quality judgment.

[0075] Furthermore, in step (8), a three-level intelligent shielding mechanism is introduced during the solder ball detection process to improve the stability and anti-interference capability of the detection. The three-level intelligent shielding includes:

[0076] First, based on the overall structure of the solder ball array, row and column level screening is performed. By analyzing the distribution of solder balls in the array, abnormal rows or columns that exceed the preset neighborhood range of the theoretical position of the solder ball array are shielded, thereby eliminating large-scale structural interference.

[0077] Secondly, local screening is performed on individual solder ball regions. By limiting the detection range to the neighborhood of the solder ball position in the template, candidate points that do not meet the spatial neighborhood constraints are eliminated to avoid false matching caused by noise or spurious features.

[0078] Finally, the detection results are masked at the region level by marking and excluding areas with severe defects or abnormal lighting in the image, thereby avoiding these areas from interfering with the overall detection results.

[0079] Furthermore, in step (9), the solder ball quality judgment is based on the center position of the solder ball and its grayscale distribution characteristics. The specific process includes:

[0080] First, based on the coordinates of the solder ball center obtained from point set registration, the centroid position of the solder ball is calculated based on image moments, which is expressed as:

[0081] ;

[0082] in, Represents the zeroth moment, and They represent the first-order moments, Indicates the coordinates of the centroid of the solder ball;

[0083] Subsequently, an inner and outer circular region is constructed with the centroid as the center, wherein the diameter of the inner circle is... With outer diameter They are defined as follows:

[0084] ;

[0085] in, Indicates the standard diameter of the solder ball. Indicates the inner circle coefficient. Let represent the external circularity coefficient, and satisfy . ;

[0086] After constructing the inner and outer circular regions, divide the circumference into a preset number of segments. Divide the circle into equal angles, and select corresponding points on the inner and outer circles in each segment direction to form a set of point pairs;

[0087] For each pair of inner and outer points, calculate their grayscale difference. A pair is considered a valid pair (ValidPair) if the following conditions are met:

[0088] ;

[0089] in, This represents the grayscale value of the inner dot. This represents the grayscale value of the outer circle. Indicates the contrast threshold;

[0090] The number of point pairs that satisfy the above conditions is denoted as . The solder ball quality score is defined as follows:

[0091] ;

[0092] in, This indicates the score for the quality of the solder balls. Indicates the total number of segments;

[0093] Finally, the quality score is compared with a quality determination threshold, and when is satisfied, the solder ball is determined as a qualified solder ball, otherwise it is determined as an unqualified solder ball, where represents the quality determination threshold.

[0094] The present invention further provides a packaged chip solder ball array quality inspection system based on perspective transformation, for implementing the above method, the system comprising:

[0095] an image acquisition module, configured to acquire an original chip image and transmit the same to an image preprocessing module;

[0096] the image preprocessing module, configured to perform nine-grid partition illumination assessment and layered compensation processing on the image, so as to achieve image homogenization and contrast enhancement;

[0097] a rough chip edge extraction module, configured to roughly extract the chip edge of the to-be-inspected image by combining an edge ROI and a multi-directional Sobel operator, so as to obtain a set of chip edge points;

[0098] a fine chip edge extraction module, configured to perform overall linear fitting on the edge points by introducing chip shape constraints, to complete fine extraction of the edges of the solder ball array, and obtain the boundary straight lines of the chip and the corresponding corner point coordinates;

[0099] a perspective correction module, configured to calculate a perspective transformation matrix based on the four corner points of the chip, and perform perspective transformation on the image, so that the chip image is converted from the original perspective view to a standard front view;

[0100] a virtual template solder ball topology and size / boundary construction module, configured to construct the solder ball topology of the virtual template according to specification parameters, and construct the size / boundary of the virtual template according to the image size and the chip position, so that the virtual template and the solder ball array in the corrected image are initially aligned;

[0101] a precise positioning module, configured to match the positions of candidate solder balls in the virtual template and the corrected image by using the topological structure feature of the solder ball array, divide a local region for each solder ball position according to the correspondence between the center points of the solder balls in the virtual template and the center points of the candidate solder balls in the corrected image, extract solder ball features within the local region, and precisely position the center position of each solder ball through a local feature optimization method;

[0102] a point set registration module, configured to complete point set registration based on centroid constraints and singular value decomposition, so as to achieve global alignment of the virtual template and the measured solder ball array;

[0103] a three-level intelligent shielding and morphological processing module, configured to extract the contour of the solder ball through three-level intelligent shielding and morphological processing;

[0104] The solder ball quality assessment module is used to calculate the solder ball quality score based on the grayscale difference between the inner and outer circles, and output the chip solder ball defect detection results and quality assessment data.

[0105] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention introduces a multi-level constraint and layer-by-layer optimization mechanism in the solder ball array detection process, and achieves significant improvements in key aspects such as boundary modeling, spatial correction, array registration and quality assessment; (2) The present invention extracts edges by combining the polarized gradient detection method in the region of interest at the edge, and further introduces a global linear fitting model based on rectangular structure constraints, transforming the traditional local edge detection problem into a global optimization problem, thereby effectively reducing the impact of noise and uneven illumination on boundary positioning and improving the stability and accuracy of corner point extraction; (3) The present invention realizes the unified mapping of chip images from perspective space to standard plane space by constructing a perspective transformation model based on corner points, which significantly reduces the impact of pose changes on solder ball positioning accuracy compared with the existing method of directly detecting on the original image; (4) The present invention introduces a two-level registration strategy of "topological constraint precise positioning + centroid SVD point set registration", firstly using the solder ball array topology and candidate point neighborhood window sub-pixel circle fitting for precise positioning, and then using decentralized SVD. Rigid transformation is precisely aligned, thereby improving the matching robustness in the case of missing solder balls or interference; (5) This invention realizes the quantitative evaluation of solder ball quality by constructing inner and outer circular structures and combining grayscale contrast mechanism. Compared with traditional methods based on single threshold or area features, it has stronger stability and discrimination ability. Attached Figure Description

[0106] Figure 1 This is an overall flowchart of the method of the present invention.

[0107] Figure 2 This is a flowchart of the fine extraction process for linear fitting of chip boundaries based on shape constraints.

[0108] Figure 3 This is a flowchart of the precise localization process based on local feature optimization and error minimization.

[0109] Figure 4 This is a flowchart of point set registration based on centroid constraints and singular value decomposition.

[0110] Figure 5 This is a flowchart for judging the quality of chip solder balls.

[0111] Figure 6 This is a diagram showing the effect of chip edge extraction.

[0112] Figure 7 The image shows the effect of virtual template registration in this invention (the left side is the unregistered image, and the right side is the registered image).

[0113] Figure 8 This image shows the quality inspection results of a solder ball array to indicate any missing balls.

[0114] Figure 9 This image shows the quality inspection results of the solder ball array, highlighting instances of ball flattening and abnormal ball roundness.

[0115] Figure 10 This image shows the quality inspection results of the solder ball array, highlighting ball offset and missing ball conditions. Detailed Implementation

[0116] To address the problems in existing technologies, such as unstable chip boundary extraction under complex lighting conditions, large positioning errors in solder ball arrays due to perspective distortion, and the lack of unified quantitative standards for solder ball quality assessment, this invention achieves high-precision positioning and stable quality detection of packaged chip solder ball arrays through multi-stage technological innovation. This includes an image preprocessing method based on partitioned lighting assessment and layered compensation; a boundary extraction method combining edge regions of interest and polarized gradient constraints; an overall linear fitting model incorporating chip shape constraints; a perspective transformation correction method based on corner points; an adaptive virtual template construction method driven by specification parameters; and precise positioning combining local feature optimization and error minimization. Then, point set registration is performed based on centroid constraints and singular value decomposition. Furthermore, a three-level intelligent shielding mechanism and a quality assessment strategy based on inner and outer circle grayscale comparison are combined to achieve high-precision positioning and stable quality detection of packaged chip solder ball arrays.

[0117] This invention proposes a machine vision-based perspective transformation-based method and system for inspecting the quality of solder ball arrays in packaged chips. The method includes: employing a nine-grid partitioning and layered compensation strategy to achieve adaptive illumination compensation, eliminating uneven illumination and reflection interference; achieving high-precision extraction of solder ball edges through edge ROI and multi-directional Sobel edge enhancement; introducing chip outer contour shape constraints to perform overall linear fitting of the solder ball array, eliminating positioning errors caused by chip tilt, offset, and deformation; achieving dynamic template mapping based on perspective transformation to correct chip posture distortion; employing a two-level registration strategy based on precise positioning using local feature optimization and error minimization, and point set registration based on centroid constraints and singular value decomposition to achieve precise alignment between the virtual template and the image under test; achieving precise template calibration through a three-level intelligent shielding mechanism and interactive fine-tuning; and finally, quantitative calculation and automatic judgment of defects such as missing, offset, flattened, and abnormal roundness based on a solder ball quality evaluation model. This invention enables high-precision and robust inspection of solder ball arrays in BGA / CSP packaged chips in industrial environments, significantly reducing the false negative and false positive rates, and is suitable for industrial online automated inspection.

[0118] Combination Figures 1-5 The method for inspecting the solder ball array quality of packaged chips based on perspective transformation in this embodiment includes the following steps:

[0119] Step 1: Image Acquisition and Preprocessing. In practice, chip images are acquired using an industrial camera, with a resolution of, for example, 1920×1080 pixels or higher. After image acquisition, the image is divided into a 3×3 grid. The average brightness value and brightness fluctuation are calculated for each area, and the illumination unevenness index is calculated, with a high threshold H set. threshold Low threshold L threshold When the unevenness index is less than L threshold At that time, it was assumed that the illumination was uniform, and when it was between L threshold To H threshold The range between H is considered moderately uneven; when it is greater than H... threshold The image was deemed severely uneven. Based on this metric, different enhancement strategies were selected. For example, 8×8 block CLAHE enhancement was used under uniform lighting conditions, while 16×16 block enhancement was used under non-uniform lighting conditions. Simultaneously, Gaussian filtering was used for smoothing in bright areas, and gamma enhancement was used in low-brightness areas to obtain an image with uniform brightness distribution.

[0120] Step 2: Edge ROI Extraction and Polarized Sobel Edge Detection. After preprocessing, regions of interest are set at the top, bottom, left, and right edges of the image, such as... Figure 6 As shown in the figure, the blue rectangle represents the region of interest, and edge detection is performed only within this region. During edge detection, the Sobel operator is used to calculate gradients, and polarity constraints are set according to the boundary directions. For example, only positive horizontal gradient responses are retained at the left edge, negative gradient responses at the right edge, and vertical gradients are used at the top and bottom edges. Simultaneously, an adaptive thresholding strategy is employed; for example, the baseline threshold can be set to 120, and dynamically adjusted within ±20% based on the region's brightness deviation, thereby obtaining a continuous and stable set of edge points.

[0121] Step 3: Linear Fitting of Chip Boundaries Based on Shape Constraints. After obtaining the set of edge points, the chip boundaries are fitted as a whole. In the specific implementation, the four sides of the chip are uniformly modeled as straight lines, and the least squares method is used to solve all edge points globally, rather than fitting each edge individually. Rectangular geometric constraints are introduced during the fitting process, such as requiring the included angle between opposite sides to be close to 180° (allowable error, e.g., ±2°), and the included angle between adjacent sides to be close to 90° (allowable error, e.g., ±2°), thus ensuring that the fitting result conforms to the rectangular structure of the chip. After the fitting is completed, the four corner points are obtained by calculating the intersection points of adjacent straight lines, and sorted in the order of top left, top right, bottom right, and bottom left to further improve the shape. Figure 6 Accuracy of extracting mid-edge lines and corner points.

[0122] Step 4: Perspective Transformation. After obtaining the chip corner points, the perspective transformation matrix is ​​calculated based on the correspondence between the corner points, and the original image is resampled to map the chip image from the tilted shooting perspective to the standard frontal view plane. In actual implementation, the output image size can be set to be consistent with the template size, such as 1000×1000 pixels, to ensure consistency in subsequent processing.

[0123] Step 5: Virtual Template Generation. A virtual solder ball topology array is generated on the perspective-corrected image based on the chip specifications. For example, when the solder ball array is a 20×20 array, a regular grid point set can be generated based on the design spacing. Each point in the template represents a theoretical solder ball center position and is used for subsequent positioning and registration. Additionally, the size and boundaries of the virtual template are constructed based on the measured dimensions and boundary conditions of the image to be inspected.

[0124] Step 6: Precise localization based on local feature optimization and error minimization. In the corrected image, candidate solder ball regions are extracted through threshold segmentation and connected component analysis, and their center positions are calculated. Then, by comparing the relative distances between template points and the distances between candidate points, topological constraints are established to obtain a set of candidate solder ball center points in the corrected image. In practical implementation, the least squares method or RANSAC method can be used to estimate the transformation parameters. A local window is constructed centered on each template point; for example, the window size can be set to 0.5 to 1 times the standard solder ball spacing. Edge extraction is performed within this window, and a sub-pixel level method is used to improve edge accuracy. Subsequently, a circular model is used to fit the solder ball contour to obtain the solder ball center coordinates.

[0125] Step 7: Point set registration based on centroid constraints and singular value decomposition. Based on the precise positioning results, globally align the template points and the measured points. First, calculate and decenter the centroids of the two point sets to eliminate the translation effect; then, solve for the optimal rigid rotation relationship through singular value decomposition, obtain the translation amount by combining the centroids, and construct a rigid transformation model to achieve precise global alignment between the template and the measured solder ball array. Figure 7 The purple rectangle on the left represents the generated virtual template, which deviates significantly from the chip under test below it. The registration effect between the virtual template and the chip under test on the right shows that, after precise positioning and point set registration, the template and the actual solder ball array can achieve completely accurate alignment without misalignment or rotational deviation. If necessary, an overall error analysis can be performed on all solder ball positions, and the registration results can be fine-tuned.

[0126] Step 8: Three-level intelligent shielding mechanism. During the solder ball inspection process, a three-level intelligent shielding strategy is introduced to improve robustness. The three-level intelligent shielding includes array-level screening, single-ball level shielding, and region-level shielding, which improves the stability of the inspection process by hierarchically filtering interference information. At the array level, abnormal rows and columns are eliminated by detecting whether the solder ball distribution conforms to a regular grid structure; at the local level, candidate solder balls are searched only within a radius of 1.2 times the template position; at the region level, regions with abnormal brightness or defects are shielded, so as to reduce false detections and missed detections.

[0127] Step 9: Morphological processing. The binarized solder ball contour image output by the three-level intelligent shielding is processed, and operations such as erosion and dilation are performed to separate adhered solder balls, remove burrs, smooth contours and fill holes, without changing the grayscale image information, without changing the center coordinates of the solder balls, and without affecting the registration accuracy and quality judgment results, so as to improve the completeness and purity of the solder ball contours and extract the precise positions of the solder balls.

[0128] Step 10: Solder ball quality judgment. After obtaining the center positions of the solder balls, the quality of the solder balls is evaluated. First, the centroid position of the solder ball is calculated, then an inner circle region and an outer circle region are constructed with the centroid as the center, wherein the diameter of the inner circle is set to, for example, 0.9 times the diameter of the solder ball, and the diameter of the outer circle is set to 1.1 times the diameter of the solder ball. It is divided into, for example, 256 equal-angle directions along the circumferential direction, the grayscale values of the corresponding positions of the inner circle and the outer circle are obtained in each direction respectively, and the difference therebetween is calculated. When the grayscale difference is greater than the contrast threshold T contrast , it is considered that the direction satisfies the clear boundary condition. The number of all directions that meet the conditions is counted, and the ratio of the number to the total number of directions is calculated as the quality score. When the quality score is greater than, for example, 80%, the solder ball is judged to be qualified, otherwise it is judged to be unqualified. For example Figure 8 ~ Figure 10 , the positions of the chip solder balls are circled with red circles in the three effect diagrams, and the situations of missing balls, flattened balls, abnormal ball roundness and ball offset are marked at the upper right corner of the circles. It can be seen from the effect diagrams that the method of the present invention can accurately identify defects such as missing balls, abnormal ball roundness and ball offset, with clear marks and reliable results.

Claims

1. A method for inspecting the quality of solder ball arrays in packaged chips based on perspective transformation, characterized in that, include: (1) Acquire the original image of the chip and perform preprocessing. Use a nine-grid partitioning and layer compensation strategy to achieve image homogenization and contrast enhancement. (2) The chip edges of the image under test are roughly extracted by using edge ROI combined with multi-directional Sobel operator to obtain the chip edge point set; (3) By introducing chip shape constraints, the edge points are linearly fitted as a whole to complete the fine extraction of the solder ball array edge and obtain the chip boundary line and the corresponding corner coordinates; (4) Based on the four corner points of the chip, calculate the perspective transformation matrix and perform perspective transformation on the image to convert the chip image from the original perspective view to the standard frontal view. (5) Construct the solder ball topology of the virtual template according to the specification parameters, and construct the virtual template size / boundary according to the image size and chip position, so that the virtual template is initially aligned with the solder ball array in the calibration image; (6) Using the topological features of the solder ball array, the virtual template and the candidate solder ball positions in the calibration image are matched. By the correspondence between the center point of the solder ball in the virtual template and the center point of the candidate solder ball in the calibration image, a local region is divided for each solder ball position. Solder ball features are extracted in the local region, and the center position of the solder ball is accurately located by the local feature optimization method. (7) Based on precise positioning, point set registration is completed based on centroid constraints and singular value decomposition to achieve global alignment between the virtual template and the measured solder ball array; (8) Extract the solder ball contour through three-level intelligent shielding and morphological processing; (9) The quality of solder balls is determined based on the gray difference between the inner and outer circles, and the chip solder ball defect detection results and quality assessment data are output.

2. The method for inspecting the quality of solder ball arrays in a packaged chip based on perspective transformation according to claim 1, characterized in that, In step (1), the image is subjected to a nine-grid partitioned illumination assessment and layered compensation processing. The nine-grid partitioning refers to dividing the image into 3×3 regions, and calculating the average brightness value, brightness standard deviation, and histogram skewness for each region. The brightness standard deviation is used to characterize the degree of brightness fluctuation within the region and to determine the intensity of local reflections and noise. The histogram skewness is used to determine the overall brightness distribution trend of the region and to identify dark and overexposed areas. An illumination unevenness index is constructed based on the average brightness value, and its calculation formula is as follows: ； in, and These represent the maximum and minimum values ​​of the average brightness for each region, respectively. This represents the overall average brightness of the image. Based on the illumination unevenness index, the image undergoes layered compensation processing. At the global layer, the image is enhanced using a contrast-limited adaptive histogram equalization algorithm, and the contrast cropping parameters are dynamically adjusted according to the illumination unevenness. At the local layer, Gaussian smoothing is applied to overexposed areas, and gamma correction is applied to underexposed areas, thereby achieving overall optimization of the image brightness distribution.

3. The method for inspecting the quality of solder ball arrays in a packaged chip based on perspective transformation according to claim 1, characterized in that, In step (2), multiple regions of interest are set in the four edge regions of the chip, and edge detection is performed in each region of interest using the polarized Sobel operator, wherein: For chip edges in different directions, the corresponding gradient calculation direction and gradient polarity are selected according to the direction of edge grayscale change, so as to retain only the gradient response that conforms to the chip boundary characteristics.

4. The method for inspecting the quality of solder ball arrays in a packaged chip based on perspective transformation according to claim 1, characterized in that, In step (3), the overall linear fitting process of the chip edge includes: representing the chip boundary as a model of multiple straight lines, each line being represented in the following form: ； in, , , They represent the first Parameters of the boundary lines; Then, using the distance from the edge point to the corresponding line as the error metric, an overall optimization objective function is constructed: ； in, Indicates the first The coordinates of the edge points Indicates that the edge point belongs to the first... The weighting coefficients of each boundary; During the optimization process, shape constraints of the rectangular structure of the chip are introduced. Under the constraints, the objective function is jointly optimized and solved to obtain the parameters of the four boundary lines that satisfy the geometric characteristics of the rectangle. The coordinates of the four corner points of the chip are obtained by calculating the intersection of two adjacent boundary lines. The corner points are obtained by solving the following system of linear equations: ； in, and This represents two adjacent boundary lines. , , They represent the boundary lines respectively. The parameters, , , They represent the boundary lines respectively. The parameters; solving the system of equations yields the point where the two lines intersect, which is the corner point, with coordinates of... .

5. The method for inspecting the quality of solder ball arrays in a packaged chip based on perspective transformation according to claim 1, characterized in that, In step (4), perspective transformation is achieved by establishing a mapping relationship between the original image coordinates and the corrected image coordinates, which is expressed as: ； in, Represents points in the original image. Indicates the corrected point. This represents the perspective transformation matrix, which maps an image from perspective space to standard plane space.

6. The method for inspecting the quality of solder ball arrays in a packaged chip based on perspective transformation according to claim 1, characterized in that, In step (5), the virtual template generation process is constructed based on the specification parameters of the packaged chip. The specification parameters include the number of rows and columns of the solder ball array and the spacing information between adjacent solder balls, thereby forming a template point set with a regular topology. Based on the specified parameters, a solder ball array model is established in a two-dimensional plane coordinate system, wherein the template is... line, number The center coordinates of the solder balls in the column are represented as follows: ； in, This indicates the center coordinates of the corresponding solder ball in the template. and These represent the row and column indices of the solder balls in the array, respectively. and These represent the spacing parameters between adjacent solder balls in the horizontal and vertical directions, respectively. The set of template points satisfies the array topology consistency constraint, that is, the distance between any two adjacent template points satisfies: ； To ensure the regularity and consistency of the template structure in space; After the template point set is constructed, a corresponding local template area is generated based on the template center point for subsequent solder ball positioning and registration processes, so that the virtual template can establish a spatial correspondence with the perspective-corrected chip image.

7. The method for inspecting the quality of solder ball arrays in a packaged chip based on perspective transformation according to claim 1, characterized in that, In step (6), the center position of the solder ball is accurately located using a local feature optimization method. The specific process includes: Step 6.1: Generate a set of virtual template solder ball center points based on the virtual template. ,in This indicates the center coordinates of the corresponding solder ball in the template, which are located at the center point of the template. Obtained by one-dimensional mapping, Indicates the number of the solder ball in the array; Step 6.2: Extract the set of candidate solder ball center points in the perspective-corrected image based on the topological relationship. These candidate solder ball center points are obtained through grayscale features, connected component analysis, or local extremum detection methods, thus forming the set of candidate solder ball center points in the corrected image. ,in This indicates the center coordinates of the corresponding solder ball in the corrected image; Step 6.3: Correct the center point of the candidate solder ball in the image. A local neighborhood window is established centered on the solder ball. The size of the selected local neighborhood window should be smaller than the standard spacing between solder balls. Simultaneously, the local neighborhood window of each solder ball is processed in a left-to-right, top-to-bottom order to ensure the uniqueness of each solder ball's edge. Within this area, the edge points of the solder ball are extracted using a sub-pixel-level edge detection method, forming the local point set corresponding to that solder ball. ,in Indicates the solder ball number. Indicates the edge point index. The image coordinates of the corresponding edge points; Step 6.4: For each solder ball's neighborhood window, fit the solder ball's contour using a circular model, expressed as follows: ； in, For local point sets Image coordinates of subpixel edge points within the area; Indicates the coordinates of the solder ball center. Indicates the radius of the solder ball; Step 6.5: Construct a least-squares error function for the set of edge points: ； The center position of the solder ball is determined by minimizing the sum of squared residuals from the edge points to the circular profile, thus obtaining the precise center coordinates of each solder ball in the corrected image. This forms a set of precise center coordinates for the solder ball array in the corrected image. ,in Indicates the solder ball number. For the first The coordinates of the center point of each solder ball.

8. The method for inspecting the quality of solder ball arrays in a packaged chip based on perspective transformation according to claim 1, characterized in that, In step (7), point set registration is completed based on centroid constraints and singular value decomposition to achieve global alignment between the virtual template and the measured solder ball array. The specific process includes: Step 7.1: Based on the center points of the virtual template solder balls and the measured center points of the solder balls, solve for the global centroid of these two sets of points respectively: ； in, For the centroid of the virtual template, Let M be the centroid of the image to be inspected, and M be the total number of solder balls. Indicates the corresponding number in the virtual template The center position of each solder ball Indicates the first in the corrected image The precise center position of each solder ball; Step 7.2: Decentralize the two sets of points to eliminate the influence of the translation components: ； ； in, For the corresponding number in the virtual template The center position of each solder ball after decentralization For the corresponding number in the image to be inspected The center position of each solder ball after decentralization For the decentralized virtual template solder ball center point set, This represents the decentralized set of candidate solder ball center points; Step 7.3: Construct the covariance matrix based on the decentralized virtual template solder ball center point set and the candidate solder ball center point set. This matrix represents the linear relationship between two sets of points and is the core basis for solving the optimal rigid rotation matrix. ； in, It is a 2×2 covariance matrix, containing the correlation features of the two sets of points; The transpose matrix of the decentralized virtual template solder ball center point set; Step 7.4: Perform singular value decomposition on the covariance matrix, decomposing the matrix into a product of an orthogonal matrix and a diagonal matrix; directly extract the optimal rigid rotation matrix from the decomposition result to ensure that the rotation transformation minimizes the matching error between the two sets of points. ； in, Represents a left singular orthogonal matrix; Represents a singular value diagonal matrix; Represents a right singular orthogonal matrix; Represents a right singular orthogonal matrix Transpose of; Step 7.5: Calculate the optimal rigid rotation matrix using the orthogonal matrix obtained from singular value decomposition. This matrix achieves optimal angular alignment between the virtual template solder ball center point set and the candidate solder ball center point set, while satisfying the orthogonality constraint of the rigid transformation: ； in, It is the transpose of the left singular orthogonal matrix U; Step 7.6: Combining the global centroids of the two point sets with the optimal rigid rotation matrix, solve for the two-dimensional global translation vector; the two-dimensional global translation vector is used to compensate for the global positional deviation of the two point sets, completing the solution for the translation components of the rigid transformation: ； in, It is a two-dimensional global translation vector, representing the overall positional offset between the virtual template solder ball center point set and the candidate solder ball center point set; Step 7.7: Combine the optimal rigid rotation matrix with the two-dimensional global translation vector to form a rigid transformation model, and transform the virtual template point set to achieve global alignment between the template point set and the measured point set, thus completing the initial registration. ； in, Indicates the aligned first A virtual template solder ball center point; Step 7.8: Output the optimal rigid transformation parameters and the globally aligned set of center points of the solder balls in the image to be inspected, and complete the point set registration of the solder ball array.

9. The method for inspecting the quality of solder ball arrays in a packaged chip based on perspective transformation according to claim 1, characterized in that, In step (8), the solder ball contour is extracted through three-level intelligent shielding and morphological processing. The three-level intelligent shielding includes: Based on the overall structure of the solder ball array, row and column level screening is performed. By analyzing the distribution of solder balls in the array, abnormal rows or columns that exceed the preset neighborhood range of the theoretical position of the solder ball array are shielded. Local screening is performed on a single solder ball area by limiting the detection range to the neighborhood of the solder ball position in the template, and candidate points that do not meet the spatial neighborhood constraints are eliminated. The detection results are masked at the regional level by marking and excluding areas in the image that have defects or abnormal lighting. In step (9), the solder ball quality judgment is based on the center position of the solder ball and its grayscale distribution characteristics. The specific process includes: First, based on the coordinates of the solder ball center obtained from point set registration, the centroid position of the solder ball is calculated based on image moments, which is expressed as: ； in, Represents the zeroth moment, and They represent the first-order moments, Indicates the coordinates of the centroid of the solder ball; Subsequently, an inner and outer circular region is constructed with the centroid as the center, wherein the diameter of the inner circle is... With outer diameter They are defined as follows: ； in, Indicates the standard diameter of the solder ball. Indicates the inner circle coefficient. Let represent the external circle coefficient, and satisfy . ; After constructing the inner and outer circular regions, divide the circumference into a preset number of segments. Divide the circle into equal angles, and select corresponding points on the inner and outer circles in each segment direction to form a set of point pairs; For each pair of inner and outer points, calculate their grayscale difference. A pair is considered valid when the following conditions are met: ； in, This represents the grayscale value of the inner dot. This represents the grayscale value of the outer circle. Indicates the contrast threshold; The number of point pairs that satisfy the above conditions is denoted as . The solder ball quality score is defined as follows: ； in, This indicates the score for the quality of the solder balls. Indicates the total number of segments; Finally, the quality score is compared with a quality determination threshold, and when is satisfied, the solder ball is determined as a qualified solder ball, otherwise it is determined as an unqualified solder ball, wherein represents the quality determination threshold.

10. A quality inspection system for solder ball arrays of packaged chips based on perspective transformation, characterized in that, The system for implementing the method according to any one of claims 1 to 9 comprises: The image acquisition module is used to acquire the raw image of the chip and transmit it to the image preprocessing module; The image preprocessing module is used to perform nine-grid partitioned illumination evaluation and layer compensation processing on the image to achieve image homogenization and contrast enhancement. The chip edge coarse extraction module is used to coarsely extract the chip edges of the image under test by combining the edge ROI with the multi-directional Sobel operator to obtain the chip edge point set; The chip edge fine extraction module introduces chip shape constraints and performs overall linear fitting on edge points to complete the fine extraction of solder ball array edges, obtaining the chip boundary line and corresponding corner coordinates; The perspective correction module calculates the perspective transformation matrix based on the four corner points of the chip, performs perspective transformation on the image, and converts the chip image from the original perspective view to the standard frontal view. The virtual template solder ball topology and size / boundary construction module is used to construct the solder ball topology of the virtual template according to the specification parameters, and to construct the virtual template size / boundary according to the image size and chip position, so that the virtual template is initially aligned with the solder ball array in the calibration image; The precise positioning module utilizes the topological features of the solder ball array to match the positions of candidate solder balls in the virtual template and the calibration image. By using the correspondence between the center point of the solder ball in the virtual template and the center point of the candidate solder ball in the calibration image, it divides each solder ball position into a local region, extracts solder ball features within the local region, and precisely locates the center position of the solder ball using a local feature optimization method. The point set registration module is used to complete point set registration based on centroid constraints and singular value decomposition, so as to achieve global alignment between the virtual template and the measured solder ball array. The three-level intelligent shielding and morphological processing module is used to extract the solder ball contour through three-level intelligent shielding and morphological processing; The solder ball quality assessment module is used to calculate the solder ball quality score based on the grayscale difference between the inner and outer circles, and output the chip solder ball defect detection results and quality assessment data.

Images