A package inspection method and system based on high dynamic range 3D imaging

Abstract

This invention discloses a packaging inspection method and system based on high dynamic range 3D imaging, relating to the field of semiconductor packaging inspection technology. The method includes: acquiring multi-exposure images of the packaged device surface; pre-classifying the point cloud into metal pads, molding compounds, and pin regions based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve; calculating the normal vector dispersion index to extract complete point cloud layers for each material; constructing a weighted spatial transformation error function using this index as dynamic weights, and fusing multiple point clouds into a high-density complete point cloud; calculating a weighted fusion value of the local surface curvature change rate and depth gradient modulus, matching it with defect feature intervals to identify cold solder joints, warpage, and cracks, and outputting their locations; calculating a dynamic threshold based on the fusion value and the normal vector dispersion index, outputting a pass / fail conclusion, and generating an inspection report. This invention significantly improves the defect detection accuracy and reliability of complex surface packaging by adaptively adjusting parameters.

Description

Technical Field

[0001] This invention relates to the field of semiconductor packaging inspection technology, and in particular to a packaging inspection method and system based on high dynamic range 3D imaging. Background Technology

[0002] During the surface mount process, packaged devices are prone to defects such as poor soldering, warping, and cracks, which directly affect the reliability and lifespan of the product. As electronic components become increasingly dense and miniaturized, higher demands are placed on the accuracy and efficiency of packaging defect detection.

[0003] Traditional 2D vision inspection struggles to acquire depth information and lacks the ability to identify 3D deformation defects such as warpage and cold solder joints. Existing 3D vision inspection methods, such as laser triangulation and structured light imaging, face the following challenges in packaging inspection: the surface of packaged devices contains various materials, including metal pads, molding compounds, and leads, with significantly different reflective properties, making it difficult to simultaneously acquire complete 3D information for each material under a single exposure condition; different materials have varying surface flatness, which can easily lead to registration errors during point cloud fusion, affecting inspection accuracy; existing methods often use fixed thresholds for defect judgment, making it difficult to adapt to surface condition fluctuations between different materials and batches, resulting in high false positive and false negative rates.

[0004] Therefore, there is an urgent need for a packaging inspection method that can adapt to the characteristics of multiple materials, achieve high-precision point cloud fusion and adaptive defect judgment. Summary of the Invention

[0005] This invention proposes a packaging inspection method based on high dynamic range 3D imaging, comprising: S1. Acquire multi-exposure images of the surface of the packaged device through high dynamic range 3D imaging. Based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve of each pixel, pre-classify the point cloud into metal pads, plastic package and pin areas, and generate three-dimensional point cloud data with pre-classification labels. S2. Calculate the normal vector dispersion index of each pre-classified region for the 3D point cloud data, and perform threshold segmentation on the index to extract the complete point cloud layer of each material and label the normal vector dispersion index. S3. Extract feature points whose normal vector dispersion rate exceeds the threshold at the boundary of each point cloud layer. Use the normal vector dispersion index as the material dynamic weight to construct a weighted spatial transformation error function. By minimizing the error, the multi-layer point cloud is fused into a high-density complete point cloud in a unified coordinate system. S4. On the fused complete point cloud, calculate the material adaptive weighted fusion value of the local surface curvature change rate and the depth gradient modulus, match the fusion value with the defect type feature interval, identify the poor weld, warping, and crack and output the location. S5. Based on the adaptive weighted fusion value and the normal vector dispersion index, calculate the dynamic adjustment threshold based on material characteristics, compare the fusion value with the threshold, output the pass / fail conclusion, and generate a report containing the defect type, location, and quantification value.

[0006] The packaging inspection method based on high dynamic range 3D imaging, as described above, involves acquiring multi-exposure images of the packaged device surface using high dynamic range 3D imaging. Based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve of each pixel, the point cloud is pre-classified into metal pads, molding compound, and pin regions, generating 3D point cloud data carrying pre-classification labels. This includes the following sub-steps: A multi-exposure image set is formed by acquiring a sequence of grayscale images of the surface of the packaged device at at least three preset exposure times using a high dynamic range 3D imaging device. For each pixel, a brightness response curve is fitted based on the change in its grayscale value at different exposure times, and the curvature coefficient of the saturation region and the dynamic range width of the linear region of the curve are calculated. Based on the preset material classification threshold, pixels are classified into metal pad areas, molding areas, or pin areas according to the curvature coefficient and dynamic range width. The classification results are then attached as labels to the corresponding 3D point cloud data to generate 3D point cloud data with material labels.

[0007] The encapsulation detection method based on high dynamic range 3D imaging, as described above, includes the following sub-steps: calculating the normal vector dispersion index of each pre-classified region from the 3D point cloud data, performing threshold segmentation on the index, extracting the complete point cloud layer of each material, and labeling the normal vector dispersion index. For 3D point cloud data with material labels, the normal vector of each point in each material region is calculated based on principal component analysis. For each material region, the angular differences of the normal vectors within the region are statistically analyzed, and the normal vector dispersion index, which characterizes the surface flatness of the region, is calculated. For different material regions, corresponding dispersion thresholds are set, and the normal vector dispersion index is segmented by thresholding to separate the complete point cloud layer and the noisy point cloud layer for each material. The average normal vector dispersion index of the region to which each point in the complete point cloud layer belongs is marked.

[0008] The encapsulation detection method based on high dynamic range 3D imaging, as described above, involves extracting feature points whose normal vector dispersion rate exceeds a threshold at the boundaries of each point cloud layer. A weighted spatial transformation error function is constructed using the normal vector dispersion index as the dynamic weight of the material. By minimizing the error, multiple point clouds are fused into a high-density complete point cloud in a unified coordinate system. This method includes the following sub-steps: Traverse the boundary regions of each complete point cloud layer, calculate the rate of change of the normal vector dispersion index between adjacent points, and mark the points whose rate of change exceeds the preset threshold as registration feature points. Using the dispersion index of the normal vector corresponding to the registration feature points as dynamic weights, a weighted spatial transformation error function is constructed with the rotation matrix and translation vector as optimization variables; By minimizing the weighted spatial transformation error function through the iterative nearest point algorithm, the optimal spatial transformation matrix is ​​solved, and the multi-layer point clouds are aligned and fused into a high-density complete point cloud in a unified coordinate system.

[0009] The encapsulation inspection method based on high dynamic range 3D imaging, as described above, involves calculating a material-adaptive weighted fusion value of the local surface curvature change rate and depth gradient modulus on the fused complete point cloud. This fusion value is then matched with defect type feature intervals to identify cold solder joints, warpage, and cracks, and their locations are output. The method includes the following sub-steps: On the fused complete point cloud, a local neighborhood is constructed for each point, and the local surface curvature change rate of that point is calculated as the local surface curvature change feature, and the depth gradient magnitude is calculated as the depth gradient change feature. Based on the material label corresponding to the point, material adaptive weights are assigned to the rate of curvature change and the depth gradient modulus, and the weighted sum is used to obtain the defect feature fusion value of the point. The defect feature fusion value is matched with the pre-constructed defect feature intervals corresponding to poor welds, warping, and cracks to identify the defect type and record the three-dimensional coordinates of the corresponding area as the defect location.

[0010] The packaging inspection method based on high dynamic range 3D imaging, as described above, involves matching defect feature fusion values ​​with pre-constructed defect feature intervals corresponding to cold solder joints, warpage, and cracks to identify the defect type and record the three-dimensional coordinates of the corresponding region as the defect location. This includes the following sub-steps: Morphological closure operations were performed on the initially identified clusters of connected points with the same type of defects to fill the voids and smooth the boundaries, resulting in continuous defect regions. Based on the geometric features of the continuous defect region, such as area, depth range, and aspect ratio, a secondary verification is performed against the preset allowable range of defect features to filter out false detection areas and confirm the final defect target.

[0011] The encapsulation inspection method based on high dynamic range 3D imaging, as described above, includes the following sub-steps: A dynamically adjusted threshold based on material properties is calculated according to the adaptive weighted fusion value and the normal vector dispersion index; the fusion value is compared with this threshold to output a pass / fail conclusion; and a report containing defect type, location, and quantification value is generated. The average normal vector dispersion index of each material region under defect-free state is statistically analyzed, and compared with the standard device calibration value to obtain the material adjustment factor. Combined with the standard defect threshold, the dynamic adjustment threshold of the corresponding material is calculated. The weighted fusion value of each point is compared with the dynamic adjustment threshold of its region. If the fusion value exceeds the threshold, it is determined to be a defective point; otherwise, it is determined to be a qualified point. Summarize the judgment results of all points, output the qualification conclusion of the packaged device, and generate an inspection report containing defect type, three-dimensional location, and quantified feature value.

[0012] This invention also proposes a packaging inspection system based on high dynamic range 3D imaging, comprising: Point cloud preprocessing and material layering module: Multi-exposure images of the packaged device surface are acquired through high dynamic range 3D imaging. Based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve of each pixel, the point cloud is pre-classified into metal pads, molding bodies and pin regions, generating 3D point cloud data with pre-classification labels. The normal vector dispersion index of each pre-classified region is calculated on the 3D point cloud data, and the index is thresholded to extract the complete point cloud layer of each material and label the normal vector dispersion index. Weighted point cloud fusion module: Extracts feature points whose normal vector dispersion rate exceeds the threshold at the boundary of each point cloud layer. Using the normal vector dispersion index as the material dynamic weight, a weighted spatial transformation error function is constructed. By minimizing the error, the multi-layer point cloud is fused into a high-density complete point cloud in a unified coordinate system. Defect identification and quality assessment module: On the fused complete point cloud, the material adaptive weighted fusion value of the local surface curvature change rate and depth gradient modulus is calculated. The fusion value is matched with the defect type feature interval to identify cold welds, warping, and cracks and output their locations. Based on the adaptive weighted fusion value and the normal vector dispersion index, a dynamic adjustment threshold based on material characteristics is calculated. The fusion value is compared with the threshold to output a pass / fail conclusion and generate a report containing the defect type, location, and quantification value.

[0013] The present invention also proposes a computer storage medium, comprising: at least one memory and at least one processor; Memory, used to store one or more program instructions; A processor for running one or more program instructions to execute a packaging inspection method based on high dynamic range 3D imaging as described above.

[0014] The beneficial effects achieved by this invention are as follows: (1) By combining high dynamic range imaging and material pre-classification with normal vector dispersion analysis, the surface characteristics of different materials can be effectively distinguished, providing more accurate input for subsequent processing. The use of material-adaptive weighted fusion and dynamic threshold determination reduces false detections and false negatives caused by differences in surface characteristics of different materials.

[0015] (2) By integrating multi-dimensional features such as local surface curvature change rate and depth gradient modulus, and matching defect feature intervals, it can effectively distinguish and accurately locate different types of defects such as cold welds, warping, and cracks.

[0016] (3) Point cloud fusion is performed based on dynamic weights of normal vector dispersion, which optimizes the alignment accuracy of multi-view or multi-layer point clouds, thereby generating a high-density and high-completeness three-dimensional model, laying the foundation for fine detection.

[0017] (4) The method and process form a complete closed loop from data acquisition, preprocessing, fusion to defect identification and judgment. It has a high degree of automation, reduces the reliance on human experience, and improves detection efficiency and consistency.

[0018] (5) The final output not only includes qualitative and location information of defects, but also provides quantitative characteristic values ​​and qualification conclusions, which facilitates quality traceability, process analysis and production optimization. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a flowchart of a packaging detection method based on high dynamic range 3D imaging provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of a packaging inspection system based on high dynamic range 3D imaging provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 like Figure 1As shown in the figure, an embodiment of this application provides a packaging inspection method based on high dynamic range 3D imaging, comprising: Step S1: Acquire multi-exposure images of the surface of the packaged device through high dynamic range 3D imaging. Based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve of each pixel, pre-classify the point cloud into metal pads, plastic package and pin areas, and generate three-dimensional point cloud data with pre-classification labels. Specifically, a high dynamic range 3D imaging device is used to acquire grayscale image sequences of the packaged device surface at at least three preset exposure times, forming a multi-exposure image set. For each pixel, a brightness response curve is fitted based on the grayscale value changes at different exposure times, and the curvature coefficient of the saturation region and the dynamic range width of the linear region are extracted from it. Based on a preset material classification threshold, the pixels are divided into metal pads, molding compounds, or pin regions by comparing the curvature coefficient and the dynamic range width. The classification results are then used as material labels and attached to the corresponding 3D point cloud data to generate 3D point cloud data with material labels. The specific steps include the following: Step S11: Acquire a sequence of grayscale images of the surface of the packaged device at at least three preset exposure times using a high dynamic range 3D imaging device to form a multi-exposure image set; The device under test (DUT) is fixed on a high-precision displacement platform, and a high dynamic range (HDR) 3D imaging system is configured. A pre-set exposure sequence consisting of at least three different exposure times, covering a wide dynamic range from low to high illumination, is used. The camera is controlled to sequentially acquire grayscale images of the DUT surface according to this exposure sequence, recording the image corresponding to each exposure time to form a multi-exposure image set. Simultaneously, coded structured light is projected onto the DUT surface using a digital projection unit, and the spatial 3D coordinates of each pixel are calculated using phase-shifting or Gray code methods to generate initial unlabeled 3D point cloud data.

[0024] Step S12: For each pixel, fit a brightness response curve based on the change in gray value at different exposure times, and calculate the curvature coefficient of the saturation region and the dynamic range width of the linear region of the curve. For each pixel in the image with the same spatial coordinates, extract the data pairs of exposure time and corresponding grayscale value from its grayscale response values ​​at different exposure times. Fit this set of data points to construct a brightness response curve function that describes the mapping relationship between exposure and output grayscale value, expressed by the following formula: ,in, Function representing the brightness response curve; This indicates the maximum grayscale value that a pixel can achieve; This represents the material mixing coefficient, with a value range of... This is used to distinguish the response characteristics of different materials; Indicates the fast response time constant; Indicates the exposure time; Indicates the oscillation amplitude coefficient; Indicates the slow response time constant; Indicates the standard deviation of noise; This represents zero-mean, unit-variance Gaussian noise.

[0025] Based on the curve function, the position of the inflection point on the curve is determined by second derivative calculation. The left side of the inflection point is the linear region, and the right side is the saturation region. Within the linear region, the average slope of the grayscale value change with exposure is calculated, and its product with the dynamic range of the grayscale value is defined as the dynamic range width of the linear region. Within the saturation region, a quadratic polynomial is locally fitted to the curve, and the second derivative of the fitted polynomial at the saturation starting point is calculated. Its absolute value is used as the curvature coefficient of the saturation region.

[0026] Step S13: Based on the preset material classification threshold, classify the pixels into metal pad area, molding body area or pin area according to the curvature coefficient and dynamic range width, and attach the classification result as a label to the corresponding three-dimensional point cloud data to generate three-dimensional point cloud data with material labels. Beforehand, typical ranges of curvature coefficients and dynamic range widths for three materials—metal pads, molding compounds, and leads—are established using standard samples, and corresponding classification thresholds are set. For each pixel, the calculated curvature coefficient in the saturation region and the dynamic range width in the linear region are used as feature vectors and input into a classifier based on a decision tree or threshold range.

[0027] If the curvature coefficient is below the first threshold and the dynamic range width is above the second threshold, it is identified as a metal pad region with high reflectivity. If the curvature coefficient is between the third and fourth thresholds and the dynamic range width is between the fifth and sixth thresholds, it is identified as a molding compound region with diffuse reflection characteristics. If the curvature coefficient is above the seventh threshold and the dynamic range width is below the eighth threshold, it is identified as a pin region with complex geometry. After pixel-level classification, the material category of each pixel is used as an additional feature dimension and associated with the corresponding three-dimensional spatial coordinates to generate a four-dimensional data structure containing x, y, z coordinates and a material label, thus forming three-dimensional point cloud data carrying pre-classified labels.

[0028] Step S2: Calculate the normal vector dispersion index of each pre-classified region of the 3D point cloud data, perform threshold segmentation on the index, extract the complete point cloud layer of each material and label the normal vector dispersion index. Specifically, principal component analysis is used to calculate the normal vector information of each sampling point in the labeled 3D point cloud data region by region, extracting the normal vector information of each sampling point in each material region. Based on this, statistical analysis of the angle differences of the normal vectors of all points within each material region is performed to construct a normal vector dispersion index to characterize the surface smoothness of the region. Then, according to the inherent differences in surface morphology of different materials, a corresponding dispersion threshold is set for each region. This threshold is used to segment the dispersion index, effectively separating the complete point cloud layer corresponding to each material from the discrete noise point cloud layer. Each point in the complete point cloud layer is assigned the average normal vector dispersion index of its respective region. The process includes the following sub-steps: Step S21: For the 3D point cloud data carrying material labels, calculate the normal vector of each point in each material region based on the principal component analysis method; For 3D point cloud data with completed material classification, each material region is processed independently. Within each material region, a local neighborhood space is established for each sampling point. The number of neighborhood points is determined by setting a fixed neighborhood radius or adaptively, and the set of spatial points surrounding the current point is selected to form the calculation sample for the covariance matrix. Principal component analysis is used to perform eigenvalue decomposition on the covariance matrix, and the eigenvector corresponding to the smallest eigenvalue is used as the estimated normal vector value for that point. To unify the direction of the normal vectors, the sensor viewpoint direction is used as a reference to adjust the direction of the normal vectors so that all normal vectors roughly point towards the sensor side. The calculation and direction calibration of the normal vectors are performed point-by-point, traversing all points within the region.

[0029] Step S22: For each material region, statistically analyze the angular differences of the normal vectors within the region, and calculate the normal vector dispersion index that characterizes the surface flatness of the region. After calculating the normal vectors of all points within each material region, the differences in normal vector angles are statistically analyzed for each region. Using the normal vectors of all points within the region as samples, the average direction of the region's normal vectors is calculated as the reference direction. The angle between the normal vector at each point and the average direction is calculated using spherical geometry methods, yielding the angle deviation distribution. Based on this, a composite dispersion index is constructed using the mean, standard deviation, and higher-order statistical characteristics of the deviation distribution to quantify the smoothness of the surface of the material region. This index is expressed by the following formula: ,in, Indicates the dispersion index of the normal vector; It represents the mean of the angle between the normal vectors, reflecting the overall bias of the normal vector direction; The standard deviation of the angle between the normal vectors measures the first-order dispersion of the angle between the normal vectors. The larger the angle, the more dispersed the distribution of the included angle; It represents a very small positive number and is used to prevent the denominator from being zero. Indicates the total number of sampling points; This indicates the traversal index, with values ​​ranging from 1 to... ; Indicates the first The angle between the normal vectors; For areas with smooth surfaces, the distribution of normal vector angles is concentrated, and the dispersion index value is low; for areas with rough surfaces or textures, the distribution of normal vector angles is dispersed, and the dispersion index value is high.

[0030] Step S23: Set corresponding dispersion thresholds for different material regions, perform threshold segmentation on the normal vector dispersion index, separate the complete point cloud layer and the noise point cloud layer of each material, and mark the average normal vector dispersion index of the region to which each point in the complete point cloud layer belongs. Based on the inherent differences in surface morphology of different materials, an independent dispersion threshold is set for each material region. The normal vector dispersion index of each point within the region is judged according to the threshold. If the dispersion index of a point is lower than or equal to the preset threshold, it is classified into the complete point cloud layer and considered a valid material surface point; if it is higher than the threshold, it is classified into the noise point cloud layer and considered a discrete noise point or anomaly point. After segmentation, for each point in the complete point cloud layer, the average normal vector dispersion index of its respective material region is assigned as the normal vector dispersion label for that point, used for surface quality assessment and defect identification in subsequent inspection processes.

[0031] Step S3: Extract feature points whose normal vector dispersion rate exceeds the threshold at the boundary of each point cloud layer. Use the normal vector dispersion index as the material dynamic weight to construct a weighted spatial transformation error function. By minimizing the error, the multi-layer point cloud is fused into a high-density complete point cloud in a unified coordinate system. Specifically, feature points whose normal vector dispersion rate exceeds a threshold are extracted from the boundary regions of each point cloud layer. Using the normal vector dispersion index as a dynamic weight reflecting material properties, a weighted spatial transformation error function is constructed. Through iterative optimization of this error function, multiple point clouds are accurately registered and fused into a high-density complete point cloud in a unified coordinate system. Specifically, the boundary range of each point cloud layer is traversed, and the change in the normal vector dispersion index between adjacent points is calculated. Points with a change rate exceeding a set threshold are selected as feature points for registration. Dynamic weight coefficients are constructed based on the normal vector dispersion index corresponding to these feature points, establishing a weighted spatial transformation error function with rotation matrix and translation vector as variables to be optimized. Then, the iterative nearest-point algorithm is used to minimize this error function to obtain the optimal spatial transformation matrix. Finally, the multiple point clouds are aligned and fused to form complete high-density point cloud data. The specific sub-steps include: Step S31: Traverse the boundary region of each complete point cloud layer, calculate the rate of change of the normal vector dispersion index between adjacent points, and mark the points whose rate of change exceeds the preset threshold as registration feature points. In each point cloud layer, the boundary contour point set is extracted using the convex hull algorithm. The K-nearest neighbor method is used to calculate the dispersion of the neighborhood normal vector for each point within the boundary region, serving as the normal vector dispersion index. The rate of change of flatness between adjacent points is calculated sequentially along the boundary, expressed by the following formula: , This indicates the rate of change in flatness between adjacent points; Indicates the first One boundary point; Indicates and Adjacent boundary points; Point The normal vector dispersion; Point The normal vector dispersion; It represents a very small positive number and is used to prevent the denominator from being zero. This represents the influence coefficient of the angle between the normal vectors, used to control the contribution weight of the change in the normal vector to the rate of change; This represents the spatial angle between two normal vectors; points whose rate of change exceeds a threshold are marked as feature points for subsequent registration.

[0032] Step S32: Using the dispersion index of the normal vector corresponding to the registration feature points as dynamic weights, construct a weighted spatial transformation error function with rotation matrix and translation vector as optimization variables; The marked feature points are used as registration key points, with each feature point corresponding to a normal vector dispersion index, reflecting local material consistency or geometric stability. Dynamic weighting coefficients are constructed using flatness characterization values ​​to give higher weight to regions with significant material changes in the registration error function, thereby constructing a weighted spatial transformation error function. The optimization variables are the rotation matrix and translation vector, expressed by the following formula: ,in, This represents the weighted spatial transformation error function; Indicates the total number of feature points; This indicates the traversal index, with values ​​ranging from 1 to... ; Indicates the dynamic weighting coefficient; express Rotation matrix; Indicates the source point cloud The three-dimensional coordinates of each feature point; express Translation vector; Indicates the target point cloud. The three-dimensional coordinates of the matching feature points; The coefficient representing the consistency constraint of the normal vector; This represents the dot product of the target normal vector and the transformed source normal vector; This represents the weight of the local structural consistency regularization term; This represents a local structural consistency regularization term.

[0033] Step S33: Minimize the weighted spatial transformation error function through the iterative nearest point algorithm, solve for the optimal spatial transformation matrix, and align and fuse the multi-layer point cloud into a high-density complete point cloud under a unified coordinate system; An iterative nearest-point algorithm is used to optimize the weighted error function. In each iteration, feature point pairs between the source and target point clouds are matched based on the current transformation matrix. The rotation matrix and translation vector that minimize the sum of squared weighted distances are solved based on dynamic weights. A closed-form solution is obtained through singular value decomposition, and the transformation parameters are updated. The iteration continues until convergence. Finally, the point clouds of each layer are transformed to a unified coordinate system, and overlapping regions are smoothed using distance-weighted averaging.

[0034] Step S4: On the fused complete point cloud, calculate the material adaptive weighted fusion value of the local surface curvature change rate and depth gradient modulus, match the fusion value with the defect type feature interval, identify the poor weld, warping, and crack and output the location; Specifically, on the fused complete point cloud, a local neighborhood is constructed for each point, the rate of change of local surface curvature and the magnitude of the depth gradient are calculated, and weighted fusion is performed by assigning adaptive weights according to the material label. The fused value is matched with the feature intervals corresponding to poor welds, warpage, and cracks to identify the defect type and record the location. Morphological closure is performed on the initially identified connected point cloud clusters of the same type of defects to obtain continuous defect regions. Then, secondary verification is performed based on geometric features such as area, depth range, and aspect ratio to filter out false detections and confirm the final defect target. The specific steps include the following: Step S41: On the fused complete point cloud, construct a local neighborhood for each point, calculate the local surface curvature change rate of the point as the local surface curvature change feature, and calculate the depth gradient magnitude as the depth gradient change feature. For the fused complete point cloud, a kd-tree-based spatial indexing structure is used to determine the local neighborhood range of each point. The neighborhood radius is adaptively set according to the local density of the point cloud, with smaller neighborhood radii used in denser regions and larger neighborhood radii appropriately expanded in sparser regions to ensure that the neighborhood contains a sufficient number of effective points. For each target point, the 3D coordinates of all points in its neighborhood are extracted, and a local plane is fitted using the least squares method. The curvature value of the point relative to the fitted plane is calculated, and the rate of change of local surface curvature is obtained by logarithmic transformation of the ratio of the variance to the mean of the curvature values ​​in the neighborhood. Simultaneously, for the depth map mapping relationship, the gradient vector of each point in the depth gradient field is calculated, and the depth gradient magnitude is obtained by taking the square root of the sum of the squares of the gradient components, reflecting the intensity of change in the depth direction.

[0035] Step S42: Based on the material label corresponding to the point, assign material adaptive weights to the rate of curvature change and the depth gradient modulus, and sum them up to obtain the defect feature fusion value of the point. Material labels for each point are read from the material classification results synchronously acquired by the imaging system. Based on pre-calibrated material sensitivity coefficients, the contribution weights of curvature change rate and depth gradient modulus are determined for different materials. For highly reflective materials, the depth gradient modulus is easily affected by noise, so its weight is appropriately reduced while the weight of curvature change rate is increased. For light-absorbing materials or regions with complex textures, the curvature change rate may fluctuate due to sampling noise, so the weight of depth gradient modulus is moderately increased. The weight coefficients are obtained through offline calibration experiments and dynamically fine-tuned based on the material recognition confidence during actual detection. Finally, the two features are weighted and summed to obtain a defect feature fusion value that incorporates material adaptability.

[0036] Step S43: Match the defect feature fusion value with the pre-constructed defect feature intervals corresponding to poor weld, warping, and cracks, identify the defect type, and record the three-dimensional coordinates of the corresponding area as the defect location; Step S431: Perform morphological closing operation on the preliminarily identified clusters of connected points with the same type of defects, fill the voids and smooth the boundaries to obtain continuous defect regions. The initially identified clusters of point clouds with the same type of defects are projected onto a two-dimensional imaging plane to construct a binary mask image. Morphological closing operations are then performed. First, dilation is applied to fill the small holes and breaks in the defect area caused by uneven point cloud sampling. Then, erosion is applied to restore the original boundary of the area and remove the boundary burrs introduced by dilation, so that the defect area is spatially continuous and has smooth boundaries, which facilitates subsequent geometric feature extraction.

[0037] Step S432: Based on the geometric features of the area, depth range, and aspect ratio of the continuous defect region, perform secondary verification with the preset allowable range of defect features to filter out false detection areas and confirm the final defect target. For the continuous defect region obtained after morphological closure, calculate its projected area, the difference between the maximum and minimum depths within the region, and the aspect ratio of the minimum bounding rectangle. Compare these geometric features with the preset allowable ranges for various types of defects. If the geometric features of a region simultaneously meet the allowable range for that type of defect, it is confirmed as the final defect target through secondary verification. If one or more geometric features exceed the allowable range, the region is judged as a false detection and filtered out. Finally, the confirmed defect target and its location information are output.

[0038] Step S5: Calculate the dynamic adjustment threshold based on material characteristics according to the adaptive weighted fusion value and the normal vector dispersion index, compare the fusion value with the threshold, output the pass / fail conclusion, and generate a report containing the defect type, location and quantification value. Specifically, the average normal vector dispersion index of different material regions under a defect-free state is statistically analyzed and compared with the calibration value of a standard device to obtain a material adjustment factor. This factor is then combined with a standard defect threshold to calculate a dynamic adjustment threshold adapted to the current material characteristics. Based on this, the weighted fusion value of each point is compared with the dynamic adjustment threshold of its corresponding region. If the fusion value exceeds the threshold, it is determined to be a defect point; otherwise, it is considered a pass point. Finally, the judgment results of all points are summarized to form an overall pass / fail conclusion for the packaged device, and an inspection report is output. This report details the defect type, three-dimensional location information, and corresponding quantitative feature values, including the following sub-steps: Step S51: Statistically calculate the average normal vector dispersion index of each material region under defect-free state, compare it with the standard device calibration value to obtain the material adjustment factor, and calculate the dynamic adjustment threshold of the corresponding material in combination with the standard defect threshold. Based on the material partition map generated in the previous steps, for each independent material region, the normal vector dispersion index of all reference points marked "defect-free" within that region is extracted. These dispersion values ​​are statistically averaged to obtain the average normal vector dispersion index of the material in the defect-free state. The corresponding material calibration values ​​of standard devices of the same model are retrieved from the standard device database, and the ratio of the two values ​​is calculated to generate the material adjustment factor. ,in Indicates the first Material adjustment factor for each material region; This indicates the preset sensitivity adjustment coefficient; Indicates the first The average normal vector dispersion of a defect-free reference point in a material region; Indicates the first Standard device calibration values ​​for each material region; Indicates the first The standard deviation of the dispersion values ​​of each material region; Indicates the first The average value of the dispersion value of each material region; This represents a very small positive number, serving to prevent the denominator from being zero. Finally, the material adjustment factor is multiplied by a preset standard defect threshold to dynamically calculate the judgment threshold applicable to the current material region, expressed by the following formula: ,in, Point The dynamic adjustment threshold at the location; Indicates the standard defect threshold; Point Belonging to Material adjustment factor for each material region; Indicates the weighting coefficient; Point Adaptive weighted fusion value The maximum gradient magnitude; Indicates the first The average gradient magnitude of all points within a material region; It represents a very small positive number.

[0039] Step S52: Compare the weighted fusion value of each point with the dynamic adjustment threshold of its region. If the fusion value exceeds the threshold, it is determined to be a defective point; otherwise, it is determined to be a qualified point. Iterate through all the sampling points to be detected within each material region, and extract the adaptive weighted fusion value calculated in the previous step for each point. This value reflects the comprehensive degree of anomaly in the three-dimensional morphology and surface properties of the point. Compare this fusion value with the dynamic adjustment threshold of the region to which the point belongs, calculated in step S51. If the fusion value is greater than the threshold, it is determined to be a defect point; otherwise, it is determined to be a qualified point.

[0040] Step S53: Summarize the judgment results of all points, output the qualification conclusion of the packaged device, and generate an inspection report containing defect type, three-dimensional location, and quantitative feature value; The system integrates the judgment results of all sampling points across all material regions and generates an overall acceptance conclusion for the device based on the presence or absence of defect points. For each point identified as a defect, its defect type is determined using a pre-defined classifier based on its material partition and the degree to which its fusion value exceeds a threshold. The final output is a structured inspection report, detailing the defect type, three-dimensional spatial coordinates (x, y, z), and quantized feature values ​​for each defect.

[0041] Example 2 like Figure 2 As shown, Embodiment 2 of this application provides a packaging inspection system based on high dynamic range 3D imaging, comprising: Point cloud preprocessing and material layering module 21: Acquires multi-exposure images of the packaged device surface using high dynamic range 3D imaging. Based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve of each pixel, the point cloud is pre-classified into metal pads, molding compounds, and pin regions, generating 3D point cloud data with pre-classification labels. The module calculates the normal vector dispersion index for each pre-classified region of the 3D point cloud data, performs threshold segmentation on this index, extracts the complete point cloud layer for each material, and labels the normal vector dispersion index. This includes the following sub-modules: Data acquisition submodule 211: Acquires grayscale image sequences of the surface of the packaged device at at least three preset exposure times using a high dynamic range 3D imaging device to form a multi-exposure image set; Material feature calculation submodule 212: For each pixel, fit the brightness response curve according to the gray value change at different exposure times, and calculate the curvature coefficient of the saturation region and the dynamic range width of the linear region of the curve. Material classification and label attachment submodule 213: Based on the preset material classification threshold, the pixel points are classified into metal pad area, molding body area or pin area according to the curvature coefficient and dynamic range width. The classification results are attached as labels to the corresponding three-dimensional point cloud data to generate three-dimensional point cloud data with material labels. Normal vector calculation submodule 214: For 3D point cloud data with material labels, calculate the normal vector of each point in each material region based on principal component analysis. Discreteness Calculation Submodule 215: For each material region, the angular differences of the normal vectors within the region are statistically analyzed, and the normal vector dispersion index, which characterizes the surface flatness of the region, is calculated. Point cloud layer extraction and labeling submodule 216: Set corresponding discreteness thresholds for different material regions, perform threshold segmentation on the normal vector discreteness index, separate the complete point cloud layer and the noise point cloud layer of each material, and label the average normal vector discreteness index of the region to which each point in the complete point cloud layer belongs. Weighted Point Cloud Fusion Module 22: Extracts feature points whose normal vector dispersion rate exceeds a threshold at the boundaries of each point cloud layer. Using the normal vector dispersion index as the dynamic weight of the material, it constructs a weighted spatial transformation error function. By minimizing the error, it fuses multiple point clouds into a high-density complete point cloud in a unified coordinate system. It includes the following sub-modules: Feature point extraction submodule 221: Traverse the boundary region of each complete point cloud layer, calculate the rate of change of the normal vector dispersion index between adjacent points, and mark the points whose rate of change exceeds the preset threshold as registration feature points; Submodule 222 for constructing weighted error function: Using the dispersion index of the normal vector corresponding to the registration feature point as the dynamic weight, a weighted spatial transformation error function is constructed with the rotation matrix and translation vector as optimization variables; Transformation optimization and fusion execution submodule 223: Minimizes the weighted spatial transformation error function through the iterative nearest point algorithm, solves the optimal spatial transformation matrix, and aligns and fuses the multi-layer point cloud into a high-density complete point cloud in a unified coordinate system; Defect Identification and Quality Judgment Module 23: On the fused complete point cloud, calculate the material adaptive weighted fusion value of the local surface curvature change rate and depth gradient modulus, match this fusion value with the defect type feature interval, identify cold welds, warpage, and cracks, and output their locations; based on the adaptive weighted fusion value and the normal vector dispersion index, calculate a dynamic adjustment threshold based on material characteristics, compare the fusion value with this threshold, output a pass / fail conclusion, and generate a report containing defect type, location, and quantification value; including the following sub-modules: Feature extraction submodule 231: On the fused complete point cloud, a local neighborhood is constructed for each point, the local surface curvature change rate of the point is calculated as the local surface curvature change feature, and the depth gradient magnitude is calculated as the depth gradient change feature. Weighted fusion submodule 232: Based on the material label corresponding to the point, assign material adaptive weights to the rate of curvature change and the depth gradient modulus, and sum the weighted values ​​to obtain the defect feature fusion value of the point; Defect type identification submodule 233: Matches the defect feature fusion value with the pre-constructed defect feature intervals corresponding to poor weld, warping, and crack, identifies the defect type, and records the three-dimensional coordinates of the corresponding area as the defect location; Dynamic threshold calculation submodule 234: Statistically calculate the average normal vector dispersion index of each material region under the defect-free state, compare it with the standard device calibration value to obtain the material adjustment factor, and calculate the dynamic adjustment threshold of the corresponding material in combination with the standard defect threshold. Qualification judgment submodule 235: Compare the weighted fusion value of each point with the dynamic adjustment threshold of the region to which it belongs. If the fusion value exceeds the threshold, it is judged as a defect point; otherwise, it is judged as a qualified point. Report generation submodule 236: Summarizes the judgment results of all points, outputs the qualification conclusion of the packaged device, and generates an inspection report containing defect type, three-dimensional location, and quantitative feature value; The memory is used to store one or more program instructions; A processor for running one or more program instructions to execute a packaging inspection method based on high dynamic range 3D imaging.

[0042] Corresponding to the above embodiments, this embodiment of the invention provides a computer-readable storage medium containing one or more program instructions, which are executed by a processor to provide a packaging detection method based on high dynamic range 3D imaging.

[0043] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed on a computer, the computer performs the aforementioned packaging detection method based on high dynamic range 3D imaging.

[0044] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0045] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0046] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0047] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0048] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0049] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0050] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0051] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A packaging inspection method based on high dynamic range 3D imaging, characterized in that, include: S1. Acquire multi-exposure images of the surface of the packaged device through high dynamic range 3D imaging. Based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve of each pixel, pre-classify the point cloud into metal pads, plastic package and pin areas, and generate three-dimensional point cloud data with pre-classification labels. S2. Calculate the normal vector dispersion index of each pre-classified region for the 3D point cloud data, and perform threshold segmentation on the index to extract the complete point cloud layer of each material and label the normal vector dispersion index. S3. Extract feature points whose normal vector dispersion rate exceeds the threshold at the boundary of each point cloud layer. Use the normal vector dispersion index as the material dynamic weight to construct a weighted spatial transformation error function. By minimizing the error, the multi-layer point cloud is fused into a high-density complete point cloud in a unified coordinate system. S4. On the fused complete point cloud, calculate the material adaptive weighted fusion value of the local surface curvature change rate and the depth gradient modulus, match the fusion value with the defect type feature interval, identify the poor weld, warping, and crack and output the location. S5. Based on the adaptive weighted fusion value and the normal vector dispersion index, calculate the dynamic adjustment threshold based on material characteristics, compare the fusion value with the threshold, output the pass / fail conclusion, and generate a report containing the defect type, location, and quantification value.

2. The packaging inspection method based on high dynamic range 3D imaging according to claim 1, characterized in that, Multi-exposure images of the packaged device surface are acquired using high dynamic range 3D imaging. Based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve of each pixel, the point cloud is pre-classified into metal pads, molding compound, and pin regions, generating 3D point cloud data with pre-classification labels. This process includes the following sub-steps: A multi-exposure image set is formed by acquiring a sequence of grayscale images of the surface of the packaged device at at least three preset exposure times using a high dynamic range 3D imaging device. For each pixel, a brightness response curve is fitted based on the change in its grayscale value at different exposure times, and the curvature coefficient of the saturation region and the dynamic range width of the linear region of the curve are calculated. Based on the preset material classification threshold, pixels are classified into metal pad areas, molding areas, or pin areas according to the curvature coefficient and dynamic range width. The classification results are then attached as labels to the corresponding 3D point cloud data to generate 3D point cloud data with material labels.

3. The packaging inspection method based on high dynamic range 3D imaging according to claim 1, characterized in that, The normal vector dispersion index of each pre-classified region in the 3D point cloud data is calculated, and the index is thresholded for segmentation. The complete point cloud layer of each material is extracted and the normal vector dispersion index is labeled. This includes the following sub-steps: For 3D point cloud data with material labels, the normal vector of each point in each material region is calculated based on principal component analysis. For each material region, the angular differences of the normal vectors within the region are statistically analyzed, and the normal vector dispersion index, which characterizes the surface flatness of the region, is calculated. For different material regions, corresponding dispersion thresholds are set, and the normal vector dispersion index is segmented by thresholding to separate the complete point cloud layer and the noisy point cloud layer for each material. The average normal vector dispersion index of the region to which each point in the complete point cloud layer belongs is marked.

4. The packaging inspection method based on high dynamic range 3D imaging according to claim 1, characterized in that, Feature points whose normal vector dispersion rate exceeds a threshold are extracted from the boundaries of each point cloud layer. Using the normal vector dispersion index as the dynamic weight of the material, a weighted spatial transformation error function is constructed. By minimizing the error, the multi-layer point cloud is fused into a high-density complete point cloud in a unified coordinate system. This includes the following sub-steps: Traverse the boundary regions of each complete point cloud layer, calculate the rate of change of the normal vector dispersion index between adjacent points, and mark the points whose rate of change exceeds the preset threshold as registration feature points. Using the dispersion index of the normal vector corresponding to the registration feature points as dynamic weights, a weighted spatial transformation error function is constructed with the rotation matrix and translation vector as optimization variables; By minimizing the weighted spatial transformation error function through the iterative nearest point algorithm, the optimal spatial transformation matrix is ​​solved, and the multi-layer point clouds are aligned and fused into a high-density complete point cloud in a unified coordinate system.

5. The package inspection method based on high dynamic range 3D imaging according to claim 1, wherein, On the fused complete point cloud, the material-adaptive weighted fusion value of the local surface curvature change rate and depth gradient modulus is calculated. This fusion value is then matched with the defect type feature interval to identify incomplete welds, warpages, and cracks, and their locations are output. This process includes the following sub-steps: On the fused complete point cloud, a local neighborhood is constructed for each point, and the local surface curvature change rate of that point is calculated as the local surface curvature change feature, and the depth gradient magnitude is calculated as the depth gradient change feature. Based on the material label corresponding to the point, material adaptive weights are assigned to the rate of curvature change and the depth gradient modulus, and the weighted sum is used to obtain the defect feature fusion value of the point. The defect feature fusion value is matched with the pre-constructed defect feature intervals corresponding to poor welds, warping, and cracks to identify the defect type and record the three-dimensional coordinates of the corresponding area as the defect location.

6. The packaging inspection method based on high dynamic range 3D imaging according to claim 5, characterized in that, The defect feature fusion value is matched with the pre-constructed defect feature intervals corresponding to poor weld, warpage, and cracks to identify the defect type and record the three-dimensional coordinates of the corresponding region as the defect location. This includes the following sub-steps: Morphological closure operations were performed on the initially identified clusters of connected points with the same type of defects to fill the voids and smooth the boundaries, resulting in continuous defect regions. Based on the geometric features of the continuous defect region, such as area, depth range, and aspect ratio, a secondary verification is performed against the preset allowable range of defect features to filter out false detection areas and confirm the final defect target.

7. The packaging inspection method based on high dynamic range 3D imaging according to claim 1, characterized in that, Based on the adaptive weighted fusion value and the normal vector dispersion index, a dynamic adjustment threshold based on material characteristics is calculated. The fusion value is compared with this threshold to output a pass / fail conclusion, generating a report containing defect type, location, and quantification value, including the following sub-steps: The average normal vector dispersion index of each material region under defect-free state is statistically analyzed, and compared with the standard device calibration value to obtain the material adjustment factor. Combined with the standard defect threshold, the dynamic adjustment threshold of the corresponding material is calculated. The weighted fusion value of each point is compared with the dynamic adjustment threshold of its region. If the fusion value exceeds the threshold, it is determined to be a defective point; otherwise, it is determined to be a qualified point. Summarize the judgment results of all points, output the qualification conclusion of the packaged device, and generate an inspection report containing defect type, three-dimensional location, and quantified feature value.

8. A package inspection system based on high dynamic range 3D imaging, characterized in that, include: Point cloud preprocessing and material layering module: Multi-exposure images of the packaged device surface are acquired through high dynamic range 3D imaging. Based on the curvature coefficient of the saturation region and the dynamic range width of the linear region of the brightness response curve of each pixel, the point cloud is pre-classified into metal pads, molding bodies and pin regions, generating 3D point cloud data with pre-classification labels. The normal vector dispersion index of each pre-classified region is calculated on the 3D point cloud data, and the index is thresholded to extract the complete point cloud layer of each material and label the normal vector dispersion index. Weighted point cloud fusion module: Extracts feature points whose normal vector dispersion rate exceeds the threshold at the boundary of each point cloud layer. Using the normal vector dispersion index as the material dynamic weight, a weighted spatial transformation error function is constructed. By minimizing the error, the multi-layer point cloud is fused into a high-density complete point cloud in a unified coordinate system. Defect identification and quality assessment module: On the fused complete point cloud, the material adaptive weighted fusion value of the local surface curvature change rate and depth gradient modulus is calculated. The fusion value is matched with the defect type feature interval to identify cold welds, warping, and cracks and output their locations. Based on the adaptive weighted fusion value and the normal vector dispersion index, a dynamic adjustment threshold based on material characteristics is calculated. The fusion value is compared with the threshold to output a pass / fail conclusion and generate a report containing the defect type, location, and quantification value.

9. A computer storage medium, comprising, include: At least one memory and at least one processor; Memory, used to store one or more program instructions; A processor for running one or more program instructions to perform a packaging inspection method based on high dynamic range 3D imaging as described in any one of claims 1-7.

Images