Chip packaging defect automatic identification method based on computer vision

Abstract

The invention discloses an automatic chip packaging defect identification method based on computer vision, which comprises the following steps of: generating multi-scale features and candidate regions, and establishing a detection head containing array parameter estimation and RepPoints point set regression and classification; the pitch, the rotation angle, the original point and the relaxation threshold value are regressed, and a pitch locking grid model is constructed; outputting a point set containing coordinates, scales, kernel weights and gating probabilities, and screening effective points; projecting according to the grid model and applying bounded offset to obtain a pitch locking point; constructing an implicit distance field based on the locking point and extracting a defect contour; calculating a minimum enclosing rectangle, a length, a width, an area and a minimum gap; fusing the classification score, the gating retention rate and the curvature stability to generate an instance confidence coefficient, and performing non-maximum suppression; and outputting defect types, position frames, contours and measurement results. According to the invention, integrated processing of array prior, contour reconstruction and metering is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial detection, and particularly relates to a chip package defect automatic identification method based on computer vision. BACKGROUND

[0002] The existing chip package defect detection adopts a combination of a convolution network and a feature pyramid, a detection head gives a position and a category, a segmentation head gives a pixel-level contour, a point set representation method such as RepPoints has been used for target geometric description, and in engineering implementation, a region of interest is usually cropped in coordination with a process coordinate, and then detection and segmentation are performed, and traditional rule methods further include frequency domain filtering, morphological operation, threshold segmentation and connected component analysis, and a candidate is screened in a classifier, and a measurement link is usually separately calculated after detection or segmentation to calculate a length, a width, an area and a minimum gap, thereby forming a loose coupling pipeline.

[0003] A number of key defects are exposed in a mass production scene, the package is in a regular array, a general detection head lacks pitch and rotation priori, a point position is easy to deviate from a grid neighborhood, false detection is significant under dense small targets, cropping and calibration depending on a process coordinate are difficult to be included in a learnable constraint, batch changes cause frequent parameter tuning, small defects and low-contrast defects are vulnerable to express under reflection and texture interference, a point set regression with a fixed number of points is difficult to simultaneously cover a fine crack and a micro hole, a point number deficiency causes a contour to be missing, and a point number redundancy brings regression instability, when detection and segmentation are separately deployed, frame-to-contour mapping relies on an additional module, errors are accumulated between modules, a length, a width, an area and a minimum gap calculation aperture are not unified, non-maximum suppression usually only relies on a classification score, and point set stability and geometric consistency are ignored, and repeated reporting and omission are prone to occur near dense solder joints.

[0004] Therefore, how to provide a chip package defect automatic identification method based on computer vision is a problem to be solved by those skilled in the art. SUMMARY

[0005] An object of the present application is to provide a chip package defect automatic identification method based on computer vision, which comprehensively uses array parameter estimation, pitch-locked grid, point set gated regression and implicit distance field reconstruction to complete integrated processing of defect positioning, contour extraction and geometric measurement, regresses a pitch, a rotation angle and an origin in a candidate region to construct a grid model, performs grid projection and bounded offset on a point set to obtain a pitch-locked point set, generates an implicit distance field from the pitch-locked point set to extract a defect contour on a zero level set, simultaneously calculates a minimum bounding rectangle, a length, a width, an area and a minimum gap, and forms an instance confidence and a non-maximum suppression output result in combination with a classification score, a gated retention rate and a curvature stability, and has the advantages of structural integration, learnable geometric priori, adaptive point number, adaptation to weak contrast small defects, unified measurement aperture, low false detection rate and simple engineering deployment.

[0006] A computer vision-based chip package defect automatic identification method according to an embodiment of the present application comprises the following steps:

[0007] An input chip package image is generated to generate a multi-scale feature map and obtain a candidate region set, and a detection head comprising an array parameter estimation branch, a RepPoints point set regression branch and a classification branch is established in each candidate region;

[0008] Pitch, rotation angle, origin and relaxation threshold are regressed by the array parameter estimation branch, and a pitch locking grid model is constructed based on the pitch, rotation angle and origin;

[0009] A point set is output by the RepPoints point set regression branch, and the point comprises two-dimensional coordinates, local scale, kernel weight and gating probability, and an effective point set is formed by screening through a gating threshold;

[0010] Grid projection is performed on the effective point set according to the pitch locking grid model, and a bounded offset constrained by the relaxation threshold is applied to each point to obtain a pitch locking point set;

[0011] An implicit distance field is constructed based on the coordinates, local scale and kernel weight of the pitch locking point set, and a defect contour is extracted at a zero level set;

[0012] A minimum bounding rectangle and geometric measurement parameters are calculated according to the defect contour, and the geometric measurement parameters include length, width, area and minimum gap;

[0013] An instance confidence is generated by combining the classification score, gating retention rate and defect contour curvature stability, non-maximum suppression is performed on the minimum bounding rectangle, and a target instance set is obtained;

[0014] The defect category, position box, defect contour and geometric measurement parameters corresponding to the target instance set are output.

[0015] Optionally, the input image generates multi-scale features and candidate regions and establishes a three-branch detection head, which specifically comprises:

[0016] Chip package images are collected, noise suppression, brightness normalization, distortion correction and size standardization are performed, and a standardized image set is obtained;

[0017] The effective field of view of the package is located in the standardized image set, and irrelevant areas are removed, and an input image is output;

[0018] The input image is scaled step by step according to a preset scale, convolution and pooling are performed in turn to obtain features at each scale, high-level features are up-sampled and fused with low-level features at the same scale layer by layer to form a multi-scale feature map;

[0019] A set of candidate regions is generated on the multi-scale feature map based on the center response and confidence threshold. Rectangular parameters are used to record the center x-coordinate, center y-coordinate, width, and height, as well as the scale layer number and spatial index.

[0020] Crop and align each candidate region on the feature map of the corresponding scale layer to obtain candidate region features of a fixed size;

[0021] A shared intermediate representation of the detection head is constructed on the candidate region features by using continuous convolution and channel compression to form a unified feature tensor, and then connecting three branches in parallel.

[0022] The array parameter estimation branch takes a shared intermediate representation as input, performs spatial averaging aggregation first, and then generates pitch, rotation angle, origin, and relaxation threshold through linear mapping. Pitch is in pixels, rotation angle is in degrees, origin is represented by candidate region coordinate system, and relaxation threshold is a non-negative scalar.

[0023] The RepPoints point set regression branch takes a shared intermediate representation as input and directly generates a point set using convolutional mapping. Each point contains two-dimensional coordinates, local scale, kernel weights, and gating probabilities. The two-dimensional coordinates are represented in the candidate region coordinate system, the local scale is in pixels and has a lower limit, the kernel weights are mapped to the range of zero to one, and the gating probabilities are mapped to the range of zero to one for point number adaptation.

[0024] The classification branch takes the shared intermediate representation as input, and sequentially performs convolution, channel weighting and linear mapping to output the category score. The output of the array parameter estimation branch, the output of the RepPoints point set regression branch and the output of the classification branch are established in a one-to-one correspondence according to the candidate region index and scale layer number.

[0025] Optionally, the array parameter estimation and pitch-locked grid model construction specifically include:

[0026] Establish a candidate region coordinate system within each candidate region, with the origin set at the center of the candidate region, the horizontal axis representing the horizontal direction of pixels, and the vertical axis representing the vertical direction of pixels.

[0027] Based on the candidate region features, directional response maps and periodic response maps are generated. The directional response maps are used for angular energy calculation, and the periodic response maps are used for pitch sampling intensity calculation.

[0028] Angle energy is scanned on the directional response map according to the angle list to obtain the angle energy curve. First, a coarse search is performed with a fixed step size to determine the angle near the energy peak. Then, a fine search with a smaller step size is performed near the peak to obtain the rotation angle.

[0029] Projecting the candidate region feature along the rotation angle direction and the orthogonal direction of the direction respectively to one dimension, calculating one-dimensional autocorrelation curves of the two projections, reading the main peak interval and performing sub-pixel peak fitting near the main peak position to obtain the pitch;

[0030] Selecting a set of strong response points in the candidate region, dividing each strong response point by the pitch along the rotation angle direction and the orthogonal direction respectively and taking the remainder, counting the centralized position of the remainder distribution, and inversely deducing the original point horizontal coordinate and the original point vertical coordinate according to the centralized position to obtain the original point;

[0031] Taking the set of strong response points as input, calculating the minimum Euclidean distance from each strong response point to the ideal grid node defined by the pitch, the rotation angle and the original point, forming a distance distribution, and taking a fixed quantile as a relaxation threshold;

[0032] Establishing two sampling directions in the rotation angle direction and the orthogonal direction with the pitch as the step, determining the horizontal index range and the vertical index range according to the candidate region boundary, enumerating the grid nodes within the index range and connecting adjacent nodes to generate row and column grid lines, and recording the pitch, the rotation angle, the original point, the relaxation threshold, the horizontal index range, the vertical index range, the grid node set and the grid line set to form a pitch-locked grid model.

[0033] Optionally, the RepPoints point set regression and the effective point set screening specifically include:

[0034] Establishing a candidate region coordinate system in each candidate region and recording the center horizontal coordinate, the center vertical coordinate, the width and the height;

[0035] Convolving, normalizing and nonlinearly mapping the candidate region feature in sequence to generate a shared intermediate representation and a point template index sequence;

[0036] Simultaneously predicting four types of quantities based on the shared intermediate representation according to the point template index, which are two-dimensional coordinate offset, local scale, kernel weight and gating probability, and the two-dimensional coordinate offset is represented in the candidate region coordinate system;

[0037] Mapping the two-dimensional coordinate offset to two-dimensional coordinates by linear scaling and center translation, with the scaling factor given by half of the width and half of the height, and the translation reference being the center horizontal coordinate and the center vertical coordinate, and the decoded two-dimensional coordinates being limited within the candidate region boundary;

[0038] Setting a lower limit for the local scale, normalizing the kernel weight to make the sum equal to one, comparing the gating probability with the gating threshold and marking the passing result;

[0039] Screening to form an effective point set according to the passing result, sorting the effective points from high to low according to the gating probability, limiting the effective point number within a preset upper limit, and recording the two-dimensional coordinates, the local scale, the kernel weight and the gating probability of the effective points.

[0040] Establish a one-to-one correspondence between the effective point set, the candidate region index, and the scale layer number.

[0041] Optionally, the generation of the pitch-locked point set based on the projection and bounded offset of the pitch-locked grid specifically includes:

[0042] Read the pitch-locked raster model and the effective point set, and establish a correlation between point numbers and candidate region indices;

[0043] For each valid point, a nearest neighbor search is performed within the set of grid nodes in the pitch-locked grid model. The grid node position is determined by the pitch, rotation angle, origin, horizontal index, and vertical index. The nearest node is recorded as the projected node, and the horizontal index and vertical index are also recorded.

[0044] Calculate the offset vector from the valid point to the projected node. The offset vector includes horizontal and vertical components.

[0045] The horizontal and vertical components of the offset vector are truncated according to the relaxation threshold to obtain a bounded offset vector.

[0046] Add the projected node position to the bounded offset vector to obtain the two-dimensional coordinates of the pitch locking point;

[0047] Perform boundary clipping on the two-dimensional coordinates of the pitch locking point so that the coordinates are within the boundary of the candidate region;

[0048] Establish a one-to-one association between the two-dimensional coordinates of the pitch locking point and the corresponding local scale, kernel weight, and gating probability to generate a set of pitch locking point attributes.

[0049] The pitch locking points are arranged in order of horizontal index priority and vertical index order to form a pitch locking point set, and the mapping relationship with the effective point set is recorded.

[0050] Optionally, the implicit distance field construction and zero-level set contour extraction specifically include:

[0051] Within the candidate region, determine the sampling range of the coverage pitch locking point set, set the grid step size, and establish a two-dimensional sampling grid;

[0052] Select the kernel function type and the equal threshold parameter. The kernel function is used to map the distance from the sampling position to the pitch locking point into weights, and the equal threshold is used to determine the zero level set.

[0053] Construct an implicit range field at the sampling location Calculate the field value at:

[0054] ;

[0055] in, It is a two-dimensional pixel coordinate vector. Let i be the two-dimensional coordinates of the i-th pitch locking point. For the first Local scale of each pitch locking point For the first The kernel weight of each pitch locking point This represents the number of pitch locking points. The field scaling constant. The field value translation constant is It is the Euclidean norm;

[0056] The field value is calculated point by point on the two-dimensional sampling grid, and the grid vertices with positive field values ​​and those with negative field values ​​are marked.

[0057] Check the vertex markers within each grid cell and mark the grid edges where the sign changes;

[0058] Connect the labeled grid edges according to the grid cell index to obtain the zero-level set curve segment;

[0059] A closed polygon is generated by splicing curve segments based on spatial adjacency relationships, and is denoted as the defect contour;

[0060] The defect contour vertex sequence is resampled and rearranged to form a defect contour dataset.

[0061] Optionally, the calculation of the minimum bounding rectangle and the generation of geometric measurement parameters specifically include:

[0062] Determine the sequence of defect contour vertices within the candidate region coordinate system, with the vertices arranged in a closed order;

[0063] The convex hull is calculated based on the defect contour, and the direction angles of the convex hull edges are extracted to form a candidate set of rotation angles.

[0064] The rotation angle candidates are processed one by one. Under a given rotation angle, the vertices of the defect profile are projected onto the axis parallel to the rotation angle and the axis perpendicular to the rotation angle respectively. The projection range of the parallel axis is calculated as the length candidate, and the projection range of the vertical axis is calculated as the width candidate. The product of the length candidate and the width candidate is used as the area candidate. The rotation angle with the smallest area candidate is selected to determine the minimum bounding rectangle.

[0065] Calculate the x-coordinate of the center, y-coordinate of the center, length, width, and rotation angle of the rectangle under the minimum bounding rectangle. Take the longer side for length and the shorter side for width.

[0066] The defect area is calculated using the directed surface accumulation addition method based on the defect contour vertex sequence. The steps are to sequentially accumulate the sum of the products of the x-coordinates of adjacent vertices and the y-coordinates of the next vertex, subtract the sum of the products of the y-coordinates of adjacent vertices and the x-coordinates of the next vertex, and then take half of the absolute value.

[0067] Based on the pitch-locked grid model, row grid lines and column grid lines are determined. For each contour point, the distance to the nearest row grid line and the distance to the nearest column grid line are calculated. The smaller value is taken as the gap value of the current contour point, and the minimum value among all the gap values ​​of the contour points is taken as the minimum gap.

[0068] The length, width, area, and minimum gap are organized into geometric measurement parameters, and a corresponding relationship is established with the candidate region index.

[0069] Optionally, the instance confidence generation and non-maximum suppression specifically include:

[0070] The classification branches generate classification scores, which are then associated one-to-one with the minimum bounding rectangle.

[0071] Calculate the gate retention rate, which is the ratio of the number of points in the valid point set to the preset upper limit of the number of points;

[0072] Curvature sequences are generated by resampling on the defect contour with equal arc lengths, the variance of the curvature sequences is calculated, and the curvature stability index is obtained through monotonic mapping.

[0073] The classification score, gate retention rate, and curvature stability index are normalized and weighted according to preset weights to generate instance confidence.

[0074] Sort the minimum bounding rectangles according to the instance confidence, set the intersection-union ratio (IU) threshold, traverse the minimum bounding rectangles in sequence, and compare the IU with the retained minimum bounding rectangles one by one. If the IU is higher than the threshold, it is marked as suppressed; if it is not higher than the threshold, it is marked as retained.

[0075] The minimum bounding rectangle is retained and its corresponding defect contour is associated with the geometric measurement parameters to form a set of target instances.

[0076] The beneficial effects of this invention are:

[0077] This invention incorporates pitch, rotation angle, and origin into the detection computation graph through array parameter estimation and pitch-locked grid model. The point set is subject to bounded offset constraints within the grid neighborhood, and the candidate position remains consistent with the geometric reference. False detections and jitter in dense weld point areas are significantly reduced. The RepPoints point set regression branch introduces gating to form an effective point set, and the number of points is adaptively allocated according to the crack and micropore morphology. Weak contrast and small defects are stably characterized. The classification branch output, gating retention rate, and curvature stability jointly participate in the instance confidence. Non-maximum suppression completes ordered deduplication in dense scenes.

[0078] This invention constructs an implicit distance field using a pitch-locked point set, directly generates defect profiles on a zero-level set, and simultaneously provides the minimum bounding rectangle and geometric measurement parameters, including length, width, area, and minimum gap, within the same detection head. The profile generation and measurement calculation are of the same origin as the point set representation, avoiding cross-network alignment errors and inconsistencies in thresholds, ensuring unified measurement standards, and generating output content in one go.

[0079] This invention maintains a single-head structure at the engineering deployment level, does not rely on external segmentation branches or offline process trimming, has a simple training and inference process, controllable parameter scale, and is suitable for stable operation and extended maintenance of production lines. Overall, it demonstrates consistent positioning and measurement, robust scoring and suppression, and is suitable for automatic identification of chip packaging defects in regular arrays. Attached Figure Description

[0080] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0081] Fig. 1 This is a flowchart of a computer vision-based automatic chip packaging defect identification method proposed in this invention;

[0082] Fig. 2 This is a schematic diagram of the detection head structure of a computer vision-based automatic chip packaging defect identification method proposed in this invention.

[0083] Fig. 3 This is a schematic diagram of pitch locking and implicit distance field generation in a computer vision-based automatic chip packaging defect identification method proposed in this invention. Detailed Implementation

[0084] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0085] refer to Figs. 1-3 An automatic identification method for chip packaging defects based on computer vision includes the following steps:

[0086] Input a chip packaging image, generate multi-scale feature maps and obtain a set of candidate regions, and build a detection head in each candidate region that includes an array parameter estimation branch, a RepPoints point set regression branch and a classification branch;

[0087] The branch regression pitch, rotation angle, origin, and relaxation threshold are estimated from the array parameters, and a pitch-locked grid model is constructed based on the pitch, rotation angle, and origin.

[0088] The point set is output from the RepPoints point set regression branch. Each point contains two-dimensional coordinates, local scale, kernel weight, and gating probability. The effective point set is formed by filtering through the gating threshold.

[0089] Perform grid projection on the effective point set according to the pitch-locked grid model, and apply a bounded offset constrained by a relaxed threshold to each point to obtain the pitch-locked point set;

[0090] An implicit distance field is constructed based on the coordinates, local scale, and kernel weight of the pitch-locked point set, and the defect contour is extracted at the zero level set.

[0091] Calculate the minimum bounding rectangle and geometric measurement parameters based on the defect profile. The geometric measurement parameters include length, width, area, and minimum gap.

[0092] By combining classification scores, gate retention rates, and defect contour curvature stability, an instance confidence score is generated. Non-maximum suppression is then applied to the minimum bounding rectangle to obtain the target instance set.

[0093] Output the defect category, location box, defect contour, and geometric measurement parameters corresponding to the target instance set.

[0094] This invention introduces array parameter estimation, RepPoints point set gated regression, and pitch-locked grid within a single detection head, and directly generates contours and unified measurements using implicit distance fields. This achieves high recall and low false detection for small defects in dense arrays, maintains stable positioning under illumination fluctuations and reflective interference, and outputs consistent results for category, location box, contour, and minimum gap of length, width, and area. The structure is simple and easy to deploy in real time on the production line.

[0095] In this embodiment, the process of generating multi-scale features and candidate regions from the input image and establishing a three-branch detection head specifically includes:

[0096] The chip packaging image is acquired, and noise suppression, brightness normalization, distortion correction and size standardization are performed to obtain a standardized image set;

[0097] The effective field of view is located and encapsulated within a standardized image set, irrelevant regions are removed, and the input image is output.

[0098] The input image is scaled up step by step according to a preset scale, and convolution and pooling are performed sequentially to obtain features at each scale. Then, the high-level features are upsampled and fused with the low-level features of the same scale layer by layer to form a multi-scale feature map.

[0099] A set of candidate regions is generated on the multi-scale feature map based on the center response and confidence threshold. Rectangular parameters are used to record the center x-coordinate, center y-coordinate, width, and height, as well as the scale layer number and spatial index.

[0100] Crop and align each candidate region on the feature map of the corresponding scale layer to obtain candidate region features of a fixed size;

[0101] A shared intermediate representation of the detection head is constructed on the candidate region features by using continuous convolution and channel compression to form a unified feature tensor, and then connecting three branches in parallel.

[0102] The array parameter estimation branch takes a shared intermediate representation as input, performs spatial averaging aggregation first, and then generates pitch, rotation angle, origin, and relaxation threshold through linear mapping. Pitch is in pixels, rotation angle is in degrees, origin is represented by candidate region coordinate system, and relaxation threshold is a non-negative scalar.

[0103] The RepPoints point set regression branch takes a shared intermediate representation as input and directly generates a point set using convolutional mapping. Each point contains two-dimensional coordinates, local scale, kernel weights, and gating probabilities. The two-dimensional coordinates are represented in the candidate region coordinate system, the local scale is in pixels and has a lower limit, the kernel weights are mapped to the range of zero to one, and the gating probabilities are mapped to the range of zero to one for point number adaptation.

[0104] The classification branch takes the shared intermediate representation as input, and sequentially performs convolution, channel weighting and linear mapping to output the category score. The output of the array parameter estimation branch, the output of the RepPoints point set regression branch and the output of the classification branch are established in a one-to-one correspondence according to the candidate region index and scale layer number.

[0105] This invention unifies input quality through standardized imaging and effective field-of-view localization, employs progressively scaled pyramid features combined with convolutional pooling and upsampling fusion to improve the discriminability of small targets, and constructs a shared intermediate representation through region cropping alignment and channel compression to achieve direct mapping and stable output of array parameters, RepPoints point sets, and classification scores under the same coordinate and indexing system.

[0106] In this embodiment, the array parameter estimation and pitch-locked grid model construction specifically include:

[0107] Establish a candidate region coordinate system within each candidate region, with the origin set at the center of the candidate region, the horizontal axis representing the horizontal direction of pixels, and the vertical axis representing the vertical direction of pixels.

[0108] Based on the candidate region features, directional response maps and periodic response maps are generated. The directional response maps are used for angular energy calculation, and the periodic response maps are used for pitch sampling intensity calculation.

[0109] Angle energy is scanned on the directional response map according to the angle list to obtain the angle energy curve. First, a coarse search is performed with a fixed step size to determine the angle near the energy peak. Then, a fine search with a smaller step size is performed near the peak to obtain the rotation angle.

[0110] The candidate region features are projected one-dimensionally along the rotation angle direction and the orthogonal direction, respectively. The one-dimensional autocorrelation curves of the two projections are calculated, the main peak spacing is read, and sub-pixel peak position fitting is performed near the main peak position to obtain the pitch.

[0111] Select a set of strong response points within the candidate region. Divide each strong response point by the pitch along the rotation angle and orthogonal direction and take the remainder. Calculate the concentration of the remainder distribution. Based on the concentration, deduce the x-coordinate and y-coordinate of the origin to obtain the origin.

[0112] Using the set of strong response points as input, calculate the minimum Euclidean distance from each strong response point to the ideal grid node defined by the pitch, rotation angle, and origin to form a distance distribution, and take a fixed quantile as the relaxation threshold;

[0113] Two sampling directions are established with the pitch as the step size in the rotation angle direction and the orthogonal direction. The horizontal index range and the vertical index range are determined according to the candidate region boundary. Within the index range, grid nodes are enumerated and adjacent nodes are connected to generate row grid lines and column grid lines. The pitch, rotation angle, origin, relaxation threshold, horizontal index range, vertical index range, grid node set and grid line set are recorded to form a pitch-locked grid model.

[0114] This invention establishes a coordinate system within the candidate region and estimates the rotation angle and pitch by jointly estimating the direction response and periodic response. It then uses remainder aggregation to back-calculate the origin and distance distribution to set a relaxation threshold, thereby constructing a pitch-locked grid model. The array geometry is embedded as a learnable constraint into the detection process, achieving point constraints, candidate localization convergence, and stable alignment under cross-batch imaging conditions. This improves the separation of dense small targets and reduces false detections caused by drift, while providing a unified reference coordinate for point set projection and measurement.

[0115] In this embodiment, the RepPoints point set regression and effective point set filtering specifically include:

[0116] Establish a candidate region coordinate system within each candidate region, and record the center x-coordinate, center y-coordinate, width, and height;

[0117] Convolution, normalization, and nonlinear mapping are sequentially performed on the candidate region features to generate a shared intermediate representation and a point template index sequence;

[0118] Based on the shared intermediate representation and point template index, four types of quantities are predicted simultaneously: two-dimensional coordinate offset, local scale, kernel weight, and gate probability. The two-dimensional coordinate offset is represented in the candidate region coordinate system.

[0119] The two-dimensional coordinate offset is mapped and decoded into two-dimensional coordinates by the center translation in a linear scale. The scale factor is given by half of the width and half of the height. The translation reference is the center horizontal coordinate and the center vertical coordinate. The decoded two-dimensional coordinates are restricted to the candidate region boundary.

[0120] Set a lower limit for local scale pruning, normalize the kernel weights to make the sum equal to one, compare the gating probability with the gating threshold and mark the passing result;

[0121] Based on the results of the filtering to form a valid point set, sort them from high to low according to the gating probability, limit the number of valid points to within the preset upper limit, and record the two-dimensional coordinates, local scale, kernel weight and gating probability of the valid points;

[0122] Establish a one-to-one correspondence between the effective point set, the candidate region index, and the scale layer number.

[0123] This invention drives RepPoints point set regression within the candidate region coordinate system using a shared intermediate representation. By proportional decoding, the offset is mapped to pixel coordinates, and scale lower bound, weight normalization, and gating threshold screening are applied to form an effective point set. This allows the number of points to be adaptively allocated according to the defect morphology, and the distribution of control points to be constrained by boundaries and correspond one-to-one with the scale layer number. As a result, stable training and inference are achieved in dense array scenarios, and the risks of redundant oscillation and under-description are reduced, while improving the accuracy of small target characterization and the consistency of instance output.

[0124] In this embodiment, the generation of the pitch-locked point set based on the projection and bounded offset of the pitch-locked grid specifically includes:

[0125] Read the pitch-locked raster model and the effective point set, and establish a correlation between point numbers and candidate region indices;

[0126] For each valid point, a nearest neighbor search is performed within the set of grid nodes in the pitch-locked grid model. The grid node position is determined by the pitch, rotation angle, origin, horizontal index, and vertical index. The nearest node is recorded as the projected node, and the horizontal index and vertical index are also recorded.

[0127] Calculate the offset vector from the valid point to the projected node. The offset vector includes horizontal and vertical components.

[0128] The horizontal and vertical components of the offset vector are truncated according to the relaxation threshold to obtain a bounded offset vector.

[0129] Add the projected node position to the bounded offset vector to obtain the two-dimensional coordinates of the pitch locking point;

[0130] Perform boundary clipping on the two-dimensional coordinates of the pitch locking point so that the coordinates are within the boundary of the candidate region;

[0131] Establish a one-to-one association between the two-dimensional coordinates of the pitch locking point and the corresponding local scale, kernel weight, and gating probability to generate a set of pitch locking point attributes.

[0132] The pitch locking points are arranged in order of horizontal index priority and vertical index order to form a pitch locking point set, and the mapping relationship with the effective point set is recorded.

[0133] This invention performs nearest neighbor projection on valid points in a pitch-locked grid model and limits the offset with a relaxed threshold to obtain a pitch-locked point set after boundary clipping. It establishes a consistent attribute mapping with local scale, kernel weight, and gating probability and organizes them in an ordered manner according to row and column indices. This keeps the points aligned with the array geometry, constrains drift, and integrates redundant distribution, thereby reducing false detections and improving the stability of contour reconstruction and measurement in dense arrays.

[0134] In this embodiment, the implicit distance field construction and zero level set contour extraction specifically include:

[0135] Within the candidate region, determine the sampling range of the coverage pitch locking point set, set the grid step size, and establish a two-dimensional sampling grid;

[0136] Select the kernel function type and the equal threshold parameter. The kernel function is used to map the distance from the sampling position to the pitch locking point into weights, and the equal threshold is used to determine the zero level set.

[0137] Construct an implicit range field at the sampling location Calculate the field value at:

[0138] ;

[0139] in, It is a two-dimensional pixel coordinate vector. Let i be the two-dimensional coordinates of the i-th pitch locking point. For the first Local scale of each pitch locking point For the first The kernel weight of each pitch locking point This represents the number of pitch locking points. The field scaling constant. The field value translation constant is For the Euclidean norm, this expression is based on approximating the non-smooth operation of minimum distance with a soft minimum formed by logarithms and exponents, on the grounds that the logarithmic-and-exponential form can aggregate the weighted Gaussian radial response of each point in a continuously differentiable manner;

[0140] The field value is calculated point by point on the two-dimensional sampling grid, and the grid vertices with positive field values ​​and those with negative field values ​​are marked.

[0141] Check the vertex markers within each grid cell and mark the grid edges where the sign changes;

[0142] Connect the labeled grid edges according to the grid cell index to obtain the zero-level set curve segment;

[0143] A closed polygon is generated by splicing curve segments based on spatial adjacency relationships, and is denoted as the defect contour;

[0144] The defect contour vertex sequence is resampled and rearranged to form a defect contour dataset.

[0145] This invention constructs a two-dimensional sampling grid within a candidate region using a pitch-locked point set as control points, forming a smooth implicit distance field. It employs logarithmic and exponential aggregation weighted kernel responses and extracts closed contours using a zero-level set. Through vertex positive and negative marking, grid edge sign changes, and sequential resampling, a regular defect contour dataset is obtained, thereby achieving continuous contour reconstruction and a unified measurement entry point, while maintaining boundary consistency and repeatability in reflective and low-contrast scenes.

[0146] In this embodiment, the calculation of the minimum bounding rectangle and the generation of geometric measurement parameters specifically include:

[0147] Determine the sequence of defect contour vertices within the candidate region coordinate system, with the vertices arranged in a closed order;

[0148] The convex hull is calculated based on the defect contour, and the direction angles of the convex hull edges are extracted to form a candidate set of rotation angles.

[0149] The rotation angle candidates are processed one by one. Under a given rotation angle, the vertices of the defect profile are projected onto the axis parallel to the rotation angle and the axis perpendicular to the rotation angle respectively. The projection range of the parallel axis is calculated as the length candidate, and the projection range of the vertical axis is calculated as the width candidate. The product of the length candidate and the width candidate is used as the area candidate. The rotation angle with the smallest area candidate is selected to determine the minimum bounding rectangle.

[0150] Calculate the x-coordinate of the center, y-coordinate of the center, length, width, and rotation angle of the rectangle under the minimum bounding rectangle. Take the longer side for length and the shorter side for width.

[0151] The defect area is calculated using the directed surface accumulation addition method based on the defect contour vertex sequence. The steps are to sequentially accumulate the sum of the products of the x-coordinates of adjacent vertices and the y-coordinates of the next vertex, subtract the sum of the products of the y-coordinates of adjacent vertices and the x-coordinates of the next vertex, and then take half of the absolute value.

[0152] Based on the pitch-locked grid model, row grid lines and column grid lines are determined. For each contour point, the distance to the nearest row grid line and the distance to the nearest column grid line are calculated. The smaller value is taken as the gap value of the current contour point, and the minimum value among all the gap values ​​of the contour points is taken as the minimum gap.

[0153] The length, width, area, and minimum gap are organized into geometric measurement parameters, and a corresponding relationship is established with the candidate region index.

[0154] This invention uses a closed vertex sequence as a basis in the candidate region coordinate system, forms a rotation angle candidate through the convex hull direction angle, and uses the projection method to determine the minimum bounding rectangle. At the same time, it calculates the area based on the directional surface accumulation addition and evaluates the minimum distance between the row and column grid lines point by point in combination with the pitch-locked grid to obtain the minimum gap. Finally, the length, width, area and minimum gap are output in a unified manner for measurement and judgment.

[0155] In this embodiment, the instance confidence generation and non-maximum suppression specifically include:

[0156] The classification branches generate classification scores, which are then associated one-to-one with the minimum bounding rectangle.

[0157] Calculate the gate retention rate, which is the ratio of the number of points in the valid point set to the preset upper limit of the number of points;

[0158] Curvature sequences are generated by resampling on the defect contour with equal arc lengths, the variance of the curvature sequences is calculated, and the curvature stability index is obtained through monotonic mapping.

[0159] The classification score, gate retention rate, and curvature stability index are normalized and weighted according to preset weights to generate instance confidence.

[0160] Sort the minimum bounding rectangles according to the instance confidence, set the intersection-union ratio (IU) threshold, traverse the minimum bounding rectangles in sequence, and compare the IU with the retained minimum bounding rectangles one by one. If the IU is higher than the threshold, it is marked as suppressed; if it is not higher than the threshold, it is marked as retained.

[0161] The minimum bounding rectangle is retained and its corresponding defect contour is associated with the geometric measurement parameters to form a set of target instances.

[0162] This invention obtains a curvature sequence by resampling the defect contour with equal arc length and obtains a curvature stability index by monotonic variance mapping. The classification score and gating retention rate are uniformly normalized with this index and weighted to form an instance confidence score. Then, ordered non-maximum suppression is performed using the cross-union threshold, so that duplicate reports and adjacent interference in the dense array are eliminated by the system, the false detection rate is reduced and the recall remains stable, and the output is a set of target instances that correspond one-to-one with the contour and measurement.

[0163] Example 1:

[0164] To verify the feasibility of this invention in practice, it was applied to the online quality inspection stage of a chip packaging production line. The goal is to automate defect identification and measurement of surface images of ball grid array packages. The inspection targets are batch-produced samples, with a single image size of 2048×2048 pixels, a solder ball pitch of approximately 0.45 mm, and a single frame containing approximately 1200 array units. Common problems encountered on-site include strong reflection, low contrast, fine and discontinuous microcracks, and texture interference caused by local contamination. Traditional processes often use template cutting + threshold segmentation or directly generate masks using segmentation networks, and then use geometric modules to calculate length, width, area, and gaps. When the array is dense and the brightness fluctuates significantly, there are many false detections and contour burrs. The measurement caliber is also prone to inconsistency with changes in threshold and post-processing.

[0165] In this scenario, the system first generates multi-scale feature maps from the acquired images and obtains a set of candidate regions. For each candidate region, a detection head is built, comprising an array parameter estimation branch, a RepPoints regression branch, and a classification branch. The array parameter estimation branch regresses the pitch, rotation angle, origin, and relaxation threshold, and constructs a pitch-locked grid model accordingly. The RepPoints regression branch outputs a point set containing two-dimensional coordinates, local scale, kernel weights, and gating probabilities. Valid point sets are then filtered out using the gating threshold. These valid point sets are then processed within the pitch-locked grid model. The process involves performing a raster projection and applying a bounded offset constrained by a relaxed threshold at each point to obtain a pitch-locked point set. Then, an implicit distance field is constructed using the coordinates, local scale, and kernel weights of the pitch-locked point set. The defect profile is extracted from the zero-level set, and the minimum bounding rectangle and geometric measurement parameters, including length, width, area, and minimum gap, are calculated accordingly. Finally, the classification score, gating retention rate, and defect profile curvature stability are weighted to form an instance confidence score. Non-maximum suppression is then performed on the minimum bounding rectangle, and the target instance set and corresponding measurement results are output.

[0166] To quantify performance, a test set of 5,000 images was selected, covering five categories of defects: solder ball defects, pin residue, bubbles, cracks, and foreign objects, all labeled using a uniform standard. Parallel comparisons were performed with three common approaches: the Mask R-CNN workflow based on segmentation, the FCOS workflow based on center-dense regression, and a combination of template matching and threshold segmentation. All methods were run on the same hardware platform with the same input size. Evaluation metrics included defect detection recall, defect classification accuracy, false positive rate, single-frame processing time, and area measurement error. The results are shown in Table 1.

[0167] Table 1. Performance Comparison of the Invention Method with Existing Detection Algorithms

[0168] Method Defect detection recall rate (%) Classification accuracy (%) False detection rate (%) Single frame processing time (ms) Area measurement error (%) Template matching and threshold segmentation 87.8 85.2 8.1 128 6.7 FCOS 93.2 91.4 4.9 141 3.9 Mask R-CNN 92.6 90.1 5.4 156 4.3 The method of the present invention 98.3 97.1 1.6 119 2.1

[0169] Statistical results show that recall and accuracy remain stable under high-density arrays, the false detection rate is significantly reduced, and the single-frame processing time remains in the hundreds of milliseconds. Further analysis of challenging scenarios reveals that when brightness fluctuations are within 20%, the false detection rate of traditional processes rises to about seven to eight percentage points, while the method of this invention remains at about two percentage points. In areas with strong reflectivity, the curvature stability index is improved to about twice that of the baseline method. For microcrack samples, the gate retention rate is increased from about 0.3 for conventional samples to about 0.5. The pitch locking point set forms a denser distribution of control points at the crack tip, and the continuity of the contour generated by the zero level set is significantly improved. Compared with manual measurement, the median area error is about two percentage points, the median length and width errors are both less than one pixel, and the minimum gap error is concentrated between one and two pixels, meeting the statistical standards for production quality inspection.

[0170] Based on comprehensive experiments and on-site verification, this invention completes array parameter estimation, pitch locking, RepPoints point set gated regression, implicit distance field profile generation, and unified measurement within a single detection head, avoiding the uncertainties caused by cross-module alignment. In common operating conditions where dense arrays and low contrast coexist, instance confidence combines classification score, gate retention rate, and curvature stability into the same scoring system. Non-maximum suppression no longer solely depends on classification score, significantly reducing duplicate reports and missed detections.

[0171] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A computer vision-based automatic chip package defect identification method, characterized in that, The method comprises the following steps: inputting a chip package image, generating a multi-scale feature map and obtaining a candidate region set, and establishing a detection head comprising an array parameter estimation branch, a RepPoints point set regression branch and a classification branch in each candidate region; regressing the pitch, rotation angle, origin and relaxation threshold value from the array parameter estimation branch, and constructing a pitch locking grid model based on the pitch, rotation angle and origin; outputting a point set from the RepPoints point set regression branch, the point comprising two-dimensional coordinates, local scale, kernel weight and gating probability, and forming an effective point set by screening through a gating threshold value; performing grid projection on the effective point set according to the pitch locking grid model, and applying a bounded offset to each point under the constraint of the relaxation threshold value to obtain a pitch locking point set; constructing an implicit distance field based on the coordinates, local scale and kernel weight of the pitch locking point set, and extracting a defect contour at the zero level set; calculating a minimum bounding rectangle and geometric measurement parameters according to the defect contour, the geometric measurement parameters comprising length, width, area and minimum gap; generating instance confidence by combining the classification score, gating retention rate and defect contour curvature stability, performing non-maximum suppression on the minimum bounding rectangle to obtain a target instance set; and outputting the defect category, position box, defect contour and geometric measurement parameters corresponding to the target instance set.

2. The computer vision-based automatic chip package defect identification method of claim 1, wherein, The method of generating a multi-scale feature and a candidate region from the input image and establishing a three-branch detection head comprises: collecting chip package images, performing noise suppression, brightness normalization, distortion correction and size standardization to obtain a standardized image set; locating the effective field of view of the package in the standardized image set, removing irrelevant areas, and outputting the input image; scaling the input image by a preset scale step by step, sequentially performing convolution and pooling to obtain features at each scale, and then upsampling the high-level features and layer-by-layer fusing with the low-level features at the same scale to form a multi-scale feature map; generating a candidate region set from the center response and confidence threshold on the multi-scale feature map, using rectangular parameters to record the center horizontal coordinate, center vertical coordinate, width, height, and recording the scale layer number and spatial index; performing cropping and alignment on each candidate region on the corresponding scale layer feature map to obtain a candidate region feature of fixed size; constructing a shared intermediate representation of the detection head on the candidate region feature, the method being continuous convolution and channel compression to form a unified feature tensor, and then parallel connecting the three branches; the array parameter estimation branch takes the shared intermediate representation as input, first performs spatial average aggregation, and then generates the pitch, rotation angle, origin and relaxation threshold value through linear mapping, the pitch being in units of pixels, the rotation angle being in units of degrees, the origin being represented in the candidate region coordinate system, and the relaxation threshold value being a non-negative scalar; the RepPoints point set regression branch takes the shared intermediate representation as input, directly generates a point set through convolution mapping, the point comprising two-dimensional coordinates, local scale, kernel weight and gating probability, the two-dimensional coordinates being represented in the candidate region coordinate system, the local scale being in units of pixels and having a lower limit, the kernel weight being mapped to the range of zero to one, and the gating probability being mapped to the range of zero to one for adaptive point number; The classification branch takes the shared intermediate representation as input, sequentially performs convolution, channel weighting and linear mapping to output a category score, and establishes a one-to-one correspondence between the array parameter estimation branch output, the RepPoints point set regression branch output and the classification branch output according to the candidate region index and the scale layer number.

3. The computer vision-based automatic chip package defect identification method of claim 1, wherein, The array parameter estimation and pitch locking grid model construction specifically comprises: A candidate region coordinate system is established in each candidate region, with the origin set at the center of the candidate region, the horizontal axis being the horizontal direction of the pixels, and the vertical axis being the vertical direction of the pixels; Based on the candidate region features, a direction response map and a period response map are generated, the direction response map being used for angle energy calculation, and the period response map being used for pitch sampling intensity calculation; Angle energy scanning is performed on the direction response map according to an angle list to obtain an angle energy curve, a coarse search is first completed with a fixed step size to determine the angle near the energy peak value, and then a fine search is performed near the peak value with a reduced step size to obtain a rotation angle; One-dimensional projections of the candidate region features are made along the rotation angle direction and the orthogonal direction of the rotation angle, respectively, one-dimensional autocorrelation curves of the two projections are calculated, a main peak pitch is read, and sub-pixel peak fitting is performed near the main peak position to obtain a pitch; A set of strong response points is selected in the candidate region, each strong response point is divided by the pitch along the rotation angle direction and the orthogonal direction, respectively, and the remainders are taken, a concentrated position of the remainder distribution is counted, and the original point is obtained by backstepping the original point horizontal coordinate and the original point vertical coordinate according to the concentrated position; With the set of strong response points as input, the minimum Euclidean distance of each strong response point to an ideal grid node defined by the pitch, the rotation angle and the original point is calculated to form a distance distribution, and a fixed quantile is taken as a relaxation threshold; Two sampling directions are established in the rotation angle direction and the orthogonal direction with the pitch as the step size, the horizontal index range and the vertical index range are determined according to the candidate region boundary, grid nodes are enumerated in the index range, adjacent nodes are connected to generate row and column grid lines, and the pitch, the rotation angle, the original point, the relaxation threshold, the horizontal index range, the vertical index range, the grid node set and the grid line set are recorded to form a pitch locking grid model.

4. The computer vision-based automatic chip package defect identification method of claim 1, wherein, The RepPoints point set regression and effective point set screening specifically comprise: A candidate region coordinate system is established in each candidate region, and the center horizontal coordinate, the center vertical coordinate, the width and the height are recorded; Convolution, normalization and non-linear mapping are sequentially performed on the candidate region features to generate a shared intermediate representation and a point template index sequence; Based on the shared intermediate representation, four types of quantities are simultaneously predicted according to the point template index, which are two-dimensional coordinate offset, local scale, kernel weight and gating probability, and the two-dimensional coordinate offset is represented in the candidate region coordinate system; The two-dimensional coordinate offset is decoded by linear proportional mapping and center translation, the proportional factor is given by half of the width and half of the height, and the translation reference is the center horizontal coordinate and the center vertical coordinate, and the decoded two-dimensional coordinate is limited within the candidate region boundary; The local scale is set to a lower limit for clipping, the kernel weight is normalized to make the sum equal to one, the gating probability is compared with a gating threshold, and the passing result is marked; According to the screening result, the effective point set is formed, the effective points are sorted from high to low according to the gating probability, the number of effective points is limited within the preset upper limit, and the two-dimensional coordinates, local scale, kernel weight and gating probability of the effective points are recorded.

5. The computer vision-based automatic chip package defect identification method of claim 1, wherein, The pitch-locked grid-based projection and bounded offset generate a pitch-locked point set, which specifically includes: Read the pitch-locked grid model and the effective point set, and establish the association between the point number and the candidate region index; Perform a nearest neighbor search for each effective point in the grid node set of the pitch-locked grid model, and determine the grid node position by pitch, rotation angle, origin, horizontal index, and vertical index. The nearest node is recorded as the projection node, and the horizontal index and vertical index are recorded. Calculate the offset vector of the effective point to the projection node, which includes the horizontal component and the vertical component. According to the relaxation threshold, the horizontal component and the vertical component of the offset vector are truncated respectively to obtain the bounded offset vector. Add the position of the projection node and the bounded offset vector to obtain the two-dimensional coordinates of the pitch-locked point. Boundary clipping is performed on the two-dimensional coordinates of the pitch-locked point to make the coordinates within the candidate region boundary. The two-dimensional coordinates of the pitch-locked point are associated with the corresponding local scale, kernel weight and gating probability to generate a pitch-locked point attribute set. Arrange the pitch-locked points according to the horizontal index priority and the vertical index order to form a pitch-locked point set, and record the mapping relationship with the effective point set.

6. The computer vision-based automatic chip package defect identification method of claim 1, wherein, The implicit distance field construction and zero level set contour extraction specifically include: Determine the sampling range covering the pitch-locked point set in the candidate region, set the grid step, and establish a two-dimensional sampling grid. Select the kernel function type and the equal value threshold parameter. The kernel function is used to map the distance from the sampling position to the pitch-locked point to the weight, and the equal value threshold is used to determine the zero level set. constructing an implicit distance field, computing a field value at a sample location at the sample location ； wherein, is a two-dimensional pixel coordinate vector, is a two-dimensional coordinate of the i-th pitch lock point, is a local scale of the i-th pitch lock point, is a local scale of the i-th pitch lock point, is a kernel weight of the i-th pitch lock point, is a kernel weight of the i-th pitch lock point, is a number of pitch lock points, is a field value scaling constant, is a field value translation constant, is a Euclidean norm; Calculate the field value point by point on the two-dimensional sampling grid, and mark the grid vertices with positive field values and negative field values. Check the vertex mark in each grid cell and mark the grid segments where the sign changes. Connect the marked grid segments according to the grid cell index to obtain the zero level set curve segment. Splice the curve segments to generate a closed polygon, which is recorded as the defect contour.

7. The computer vision-based automatic chip package defect identification method of claim 1, wherein, The minimum bounding rectangle calculation and geometric quantity measurement parameter generation specifically include: Determine the sequence of defect contour vertices in the candidate region coordinate system, and arrange the vertices in closed order. Calculate the convex hull based on the defect contour, and extract the direction angle of the convex hull edge to form a rotation angle candidate set. Process each rotation angle candidate, project the defect contour vertices to the axis parallel to the rotation angle and the axis perpendicular to the rotation angle respectively at a given rotation angle, calculate the parallel axis projection range as the length candidate, calculate the vertical axis projection range as the width candidate, take the product of the length candidate and the width candidate as the area candidate, and select the rotation angle with the smallest area candidate to determine the minimum bounding rectangle. Calculate the rectangular center horizontal coordinate, center vertical coordinate, length, width and rotation angle under the minimum bounding rectangle, and take the longer side as the length and the shorter side as the width. Calculate the defect area by the directed area accumulation method based on the sequence of defect contour vertices. The pitch locking grid model is used to determine the row grid lines and the column grid lines, the distance from each contour point to the nearest row grid line and the distance from each contour point to the nearest column grid line are calculated, the smaller value is taken as the gap value of the current contour point, and the minimum value in the gap values of all contour points is taken as the minimum gap.

8. The computer vision-based automatic chip package defect identification method of claim 1, wherein, The instance confidence generation and non-maximum suppression specifically comprises: a classification score is generated by the classification branch and is one-to-one associated with the minimum bounding rectangle; a gating retention rate is calculated, the gating retention rate being the ratio of the number of points in the effective point set to the preset upper limit of the number of points; a curvature sequence is generated by equal arc length resampling on the defect contour, the variance of the curvature sequence is calculated, and the curvature stability index is obtained through monotonic mapping; the classification score, the gating retention rate and the curvature stability index are normalized, and are combined by weighting according to a preset weight to generate an instance confidence; the minimum bounding rectangles are sorted according to the instance confidence, a threshold of intersection over union is set, the minimum bounding rectangles are sequentially traversed, and the intersection over union of each minimum bounding rectangle is compared with the retained minimum bounding rectangles one by one, the minimum bounding rectangle with the intersection over union higher than the threshold is marked as suppressed, and the minimum bounding rectangle with the intersection over union not higher than the threshold is marked as retained; the retained minimum bounding rectangle and the corresponding defect contour are associated with the geometric metrology parameters to form a target instance set.

Images