Chip solder joint defect detection method and system based on optical characteristics

Abstract

This invention relates to the field of chip solder joint defect detection technology, and provides a chip solder joint defect detection method and system based on optical properties. The method includes: acquiring a flip chip to be tested; scanning multiple solder joint regions within the fixed chip to obtain a chip grayscale image set; performing the following operations on each chip grayscale image in the set: segmenting the solder joint region to obtain an optimized binary edge image; extracting image features from the corrected chip grayscale image to obtain a feature vector; summarizing the feature vectors to obtain a feature vector set corresponding to the chip grayscale image set; normalizing the feature vector set to obtain a normalized feature vector set; constructing a solder joint defect identification model; and inputting the normalized feature vector set into the solder joint defect identification model to obtain a set of normal solder joints and a set of defective solder joints. This invention can improve the accuracy and efficiency of chip solder joint defect detection.

Description

Technical Field

[0001] This invention relates to the field of chip solder joint defect detection technology, and in particular to a chip solder joint defect detection method and system based on optical properties. Background Technology

[0002] Optical properties refer to the physical response of solder joint materials and structures to light. Chip solder joints are the interconnection points in advanced packaging forms such as flip chips, BGA, and CSP.

[0003] Currently, solder joint inspection mainly relies on manual visual inspection, traditional AOI (Automated Optical Inspection), and X-ray inspection. Manual visual inspection is inefficient, has a high rate of missed detections, and is difficult to adapt to micron-level solder joints. Conventional automated optical inspection can only identify surface morphology and is sensitive to metal reflections, shadows, and hidden defects at the bottom, easily leading to false positives and false negatives. X-ray and 3DCT equipment are expensive, slow, have low recognition rates for planar defects, and pose radiation safety hazards. As solder joint sizes shrink and spacing continues to decrease, existing inspection methods struggle to balance accuracy, speed, and safety, failing to meet the demands of high-speed production lines for non-contact, high-precision, and full-coverage defect inspection. Therefore, improving the accuracy and efficiency of chip solder joint defect inspection is an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a chip solder joint defect detection method based on optical properties and a computer-readable storage medium, the main purpose of which is to improve the accuracy and efficiency of chip solder joint defect detection.

[0005] To achieve the above objectives, the present invention provides a chip solder joint defect detection method based on optical properties, comprising:

[0006] Obtain the flip chip to be tested, identify the chip solder joint array surface based on the flip chip to be tested, and fix the flip chip to be tested based on the chip solder joint array surface to obtain a fixed chip, wherein the fixed chip includes multiple solder joint areas;

[0007] The illumination source and chip scanning path are determined. Based on the illumination source and chip scanning path, multiple solder joint areas in the fixed chip are scanned to obtain a set of chip grayscale images, in which the chip grayscale images correspond one-to-one with the solder joint areas.

[0008] Perform the following operations on each chip grayscale image in the chip grayscale image set:

[0009] Solder joint region segmentation is performed on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image;

[0010] Create a blank mask, and fill the solder joint contours on the blank mask based on the optimized binary edge image to obtain the solder joint mask image;

[0011] Image feature extraction is performed on the grayscale image of the calibration chip based on the solder joint mask image to obtain the feature vector;

[0012] By summarizing the feature vectors, we obtain the feature vector set corresponding to the chip grayscale image set;

[0013] The feature vector set is normalized to obtain the normalized feature vector set. A solder joint defect identification model is constructed by inputting the normalized feature vector set into the solder joint defect identification model to obtain the normal solder joint set and the defective solder joint set.

[0014] Based on normal solder joint sets and defective solder joint sets, chip solder joint defect detection is performed based on optical characteristics.

[0015] Optionally, the step of performing solder joint region segmentation on the chip grayscale image to obtain an optimized binary edge image includes:

[0016] Median filtering is performed on the chip grayscale image to obtain a denoised chip grayscale image. Histogram equalization is then performed on the denoised chip grayscale image to obtain an equalized chip grayscale image.

[0017] Image correction is performed on the equalized chip grayscale image to obtain a corrected chip grayscale image, wherein the corrected chip grayscale image includes multiple grayscale pixels;

[0018] Binary edge images are obtained by acquiring grayscale images from correction chips, and morphological closing operations are performed on the binary edge images to obtain optimized binary edge images.

[0019] Optionally, the step of obtaining a binary edge image based on the grayscale image of the correction chip includes:

[0020] Gray pixels are extracted sequentially from the grayscale image of the calibration chip. The grayscale pixel values ​​are determined based on the extracted grayscale pixels. The adjacent horizontal grayscale pixel values ​​and adjacent vertical grayscale pixel values ​​are determined from the grayscale image of the calibration chip based on the grayscale pixel values.

[0021] By subtracting the grayscale pixel value from the adjacent horizontal grayscale pixel value and the adjacent vertical grayscale pixel value respectively, we can obtain the gradient value in the horizontal axis direction and the gradient value in the vertical axis direction.

[0022] Calculate the pixel gradient magnitude based on the gradient values ​​in the horizontal and vertical directions, obtain the gradient threshold, and compare the pixel gradient magnitude with the gradient threshold.

[0023] If the pixel gradient magnitude is greater than or equal to the gradient threshold, the extracted grayscale pixel is taken as the edge pixel of the solder joint.

[0024] If the pixel gradient magnitude is less than the gradient threshold, the extracted grayscale pixel will be used as the background pixel.

[0025] The solder joint edge pixels and background pixels are summarized separately to obtain the solder joint edge pixel set and the background pixel set. The binary edge image is then identified based on the solder joint edge pixel set and the background pixel set.

[0026] Optionally, the image feature extraction operation on the grayscale image of the correction chip based on the solder joint mask image to obtain a feature vector includes:

[0027] Gray pixels are extracted sequentially from the grayscale image of the calibration chip. The extracted gray pixels are used as gray pixels to be calculated. The target mask value is determined from the solder joint mask image based on the gray pixels to be calculated.

[0028] If the target mask value is equal to the preset black pixel value, then the target mask value is used to replace the pixel value of the grayscale pixel to be processed, and the updated grayscale pixel is obtained.

[0029] If the target mask value is not equal to the black pixel value, then the grayscale pixel to be calculated is used as the solder joint grayscale pixel;

[0030] The updated grayscale pixels and solder joint grayscale pixels are summarized separately to obtain the updated grayscale pixel set and the solder joint grayscale pixel set. The image to be detected is identified based on the updated grayscale pixel set and the solder joint grayscale pixel set.

[0031] Geometric and grayscale feature parameters are calculated based on the image to be detected. The geometric and grayscale feature parameters are then integrated to obtain a feature vector.

[0032] Optionally, the calculation of geometric feature parameters and grayscale feature parameters based on the image to be detected includes:

[0033] Based on the image to be detected, the region of the solder joint to be tested and the set of image pixel values ​​are obtained. The minimum bounding rectangle and the outline of the solder joint region are obtained based on the region of the solder joint to be tested. The area of ​​the solder joint is calculated based on the region of the solder joint to be tested.

[0034] Obtain the rectangle width and matrix height based on the minimum bounding rectangle, and calculate the aspect ratio based on the rectangle width and matrix height;

[0035] The perimeter of the outline is calculated based on the weld point area, and the roundness is calculated based on the perimeter of the outline and the area of ​​the weld point.

[0036] The mean value of the solder joint area is calculated based on the image pixel value set, and the gray value variance of the solder joint area is calculated based on the mean value of the solder joint area.

[0037] The maximum and minimum gray values ​​are determined based on the image pixel value set. The difference between the maximum and minimum gray values ​​is calculated to obtain the gray peak value.

[0038] The number of energy pixels is counted based on the image pixel value set and the preset pixel threshold. The energy is calculated based on the number of energy pixels and the solder joint area. The entropy value is calculated based on the energy.

[0039] Geometric feature parameters were determined based on the solder joint area, aspect ratio, and roundness. Gray-scale feature parameters were determined based on the mean gray-scale value, variance gray-scale value, peak-to-peak gray-scale value, energy, and entropy value of the solder joint area.

[0040] Optionally, the formula for calculating the roundness is as follows:

[0041] ;

[0042] in, Indicates roundness. Indicates the perimeter of the outline. Represents pi (π). This indicates the area of ​​the solder joint.

[0043] Optionally, the construction of the weld joint defect identification model includes:

[0044] Obtain the training and testing sample sets, and set the BP network weight set and error precision;

[0045] The training sample set and the BP network weight set are both input into the pre-constructed BP network input layer to obtain the network output value set, which includes multiple network output values ​​and corresponds one-to-one with the training samples.

[0046] A set of expected values ​​for samples is obtained based on the training sample set, wherein each expected value corresponds one-to-one with a training sample.

[0047] The error index function value is calculated based on the sample expected value set and the network output value set.

[0048] If the error index function value is not less than the error accuracy, then calculate the Jacobian matrix and the error vector, and calculate the weight increment based on the Jacobian matrix and the error vector;

[0049] The weights of each BP network in the BP network weight set are updated using weight increments to obtain an updated BP network weight set. The updated BP network weight set is then used as the BP network weight set, and the process returns to the step of inputting both the training sample set and the BP network weight set into the pre-constructed BP network input layer.

[0050] If the error index function value is less than the error precision, then the updated BP network weight set is taken as the optimal BP network weight set.

[0051] The LM-BP network was identified based on the optimal BP network weight set. The LM-BP network was then tested using a test sample set to obtain a solder joint defect identification model.

[0052] Optionally, obtaining the training sample set and the test sample set includes:

[0053] Obtain a chip sample set, identify the known normal solder joint image set and the known defective solder joint image set based on the chip sample set, and obtain the chip sample feature vector set based on the known normal solder joint image set and the known defective solder joint image set.

[0054] The chip sample feature vector set is divided to obtain the training sample set and the test sample set.

[0055] Optionally, the formula for calculating the error index function value is as follows:

[0056] ;

[0057] in, This represents the value of the error index function. This indicates the number of training samples in the training sample set. Represents the first value in the sample expectation set. Expected value of each sample Indicates the first value in the network output value set. Each network output value, This represents the L2 norm.

[0058] To achieve the above objectives, the present invention also provides a chip solder joint defect detection system based on optical properties, comprising:

[0059] The chip fixing module is used to acquire the flip chip under test, identify the chip solder joint array surface based on the flip chip under test, and fix the flip chip under test based on the chip solder joint array surface to obtain a fixed chip. The fixed chip includes multiple solder joint areas.

[0060] The optical image acquisition module is used to determine the illumination light source and the chip scanning path. Based on the illumination light source and the chip scanning path, multiple solder joint areas in the fixed chip are scanned to obtain a set of chip grayscale images, in which the chip grayscale images correspond one-to-one with the solder joint areas.

[0061] The image feature extraction module performs the following operations on each chip grayscale image in the chip grayscale image set: performs solder joint region segmentation on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image; creates a blank mask; fills the solder joint contour on the blank mask based on the optimized binary edge image to obtain a solder joint mask image; and performs image feature extraction on the corrected chip grayscale image based on the solder joint mask image to obtain a feature vector.

[0062] The chip defect identification module is used to summarize feature vectors to obtain the feature vector set corresponding to the chip grayscale image set. The feature vector set is normalized to obtain the normalized feature vector set. A solder joint defect identification model is constructed. The normalized feature vector set is input into the solder joint defect identification model to obtain the normal solder joint set and the defective solder joint set. Based on the normal solder joint set and the defective solder joint set, the chip solder joint defect detection based on optical characteristics is completed.

[0063] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0064] Memory, storing at least one instruction;

[0065] The processor executes the instructions stored in the memory to implement the above-described method for detecting chip solder joint defects based on optical characteristics.

[0066] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned method for detecting chip solder joint defects based on optical characteristics.

[0067] To address the problems described in the background art, this invention acquires a flip chip under test, identifies the chip solder joint array surface based on the flip chip, and fixes the flip chip under test based on the chip solder joint array surface to obtain a fixed chip. The fixed chip includes multiple solder joint areas. By fixing the flip chip under test, this invention ensures that the chip maintains a stable position during the testing process, avoiding image acquisition deviations caused by displacement. The invention also determines the illumination light source and chip scanning path, and scans multiple solder joint areas in the fixed chip based on the illumination light source and chip scanning path to obtain a chip grayscale image set. Each chip grayscale image corresponds one-to-one with a solder joint area. This invention ensures consistent illumination conditions and scanning methods, giving the acquired chip grayscale image set a unified imaging standard and improving the reliability and consistency of image data. For each chip grayscale image in the chip grayscale image set, the following operation is performed: solder joint area segmentation is performed on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image. This invention, by optimizing the binary edge image, can accurately extract the solder joint contour and remove background and noise interference, highlighting the effective area of ​​the solder joint. This invention creates a blank mask and fills the solder joint contours of the blank mask based on an optimized binary edge image to obtain a solder joint mask image. This allows for accurate differentiation between the solder joint area and the background area, effectively shielding the influence of irrelevant areas on subsequent processing. Based on the solder joint mask image, image feature extraction is performed on the corrected chip grayscale image to obtain feature vectors. This invention can completely preserve the geometric and grayscale features of the solder joints while removing background interference, improving the accuracy and specificity of feature extraction. The feature vectors are summarized to obtain a feature vector set corresponding to the chip grayscale image set. The feature vector set is normalized to obtain a normalized feature vector set. A solder joint defect identification model is constructed, and the normalized feature vector set is input into the solder joint defect identification model to obtain a set of normal solder joints and a set of defective solder joints. This invention summarizes and normalizes the extracted feature vectors, eliminating dimensional differences and numerical deviations between different features, improving data uniformity and model input stability. Based on the set of normal solder joints and the set of defective solder joints, chip solder joint defect detection based on optical characteristics is completed. Therefore, the present invention can improve the accuracy and efficiency of chip solder joint defect detection. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating a chip solder joint defect detection method based on optical properties according to an embodiment of the present invention.

[0069] Figure 2 A functional block diagram of a chip solder joint defect detection system based on optical properties provided in an embodiment of the present invention;

[0070] Figure 3This is a schematic diagram of an electronic device that implements the chip solder joint defect detection method based on optical properties, according to an embodiment of the present invention.

[0071] Explanation of reference numerals in the attached figures:

[0072] 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0073] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0074] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0075] This application provides a method for detecting chip solder joint defects based on optical characteristics. The execution entity of the method includes, but is not limited to, at least one electronic device that can be configured to execute the method provided in this application, such as a server or a terminal. In other words, the method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0076] Reference Figure 1 The diagram shown is a flowchart illustrating a chip solder joint defect detection method based on optical properties according to an embodiment of the present invention. In this embodiment, the chip solder joint defect detection method based on optical properties includes:

[0077] S1. Obtain the flip chip to be tested, identify the chip solder joint array surface based on the flip chip to be tested, and fix the flip chip to be tested based on the chip solder joint array surface to obtain a fixed chip, wherein the fixed chip includes multiple solder joint areas.

[0078] It should be explained that the flip chip under test (DUT) is the finished flip chip to be inspected for solder joint defects. The chip solder joint array surface is the chip surface on the DUT where all solder joints, arranged in a regular array, are located to achieve electrical and mechanical connections. The chip fixation based on the chip solder joint array surface involves using the chip solder joint array surface as a positioning reference, placing the DUT on the inspection station or stage, ensuring the chip solder joint array surface faces the inspection optical path, and then using vacuum adsorption to fix the DUT. The fixed chip is the DUT that has already undergone the fixation process. The solder joint area is a single solder joint on the fixed chip and its corresponding surrounding imaging area.

[0079] For example, there is a SAM image of a flip chip, in which a solder joint has a diameter of 135μm, a solder joint spacing of 254μm, and an image resolution of 1μm / pixel. The solder joint diameter in the SAM image includes 135 pixels, and the solder joint spacing in the SAM image includes 254 pixels. Taking 1.3 times the solder joint diameter as the side length of the clipping window, the side length of the clipping window is 1.3 multiplied by 135 pixels, which is approximately equal to 175 pixels. Therefore, the imaging area corresponding to each solder joint is a square window of 175 pixels × 175 pixels.

[0080] It should be noted that if the flip chip under test is displaced, tilted, or shaken during the testing process, it will cause deviations in the optical path illumination angle, misalignment of the scanning path, and blurry imaging, directly affecting the quality of solder joint image acquisition and the accuracy of defect detection. Therefore, this invention fixes the flip chip under test to ensure that the position and orientation of the flip chip under test remain stable throughout the scanning and imaging process, thereby achieving accurate scanning and clear imaging of each solder joint area and improving the accuracy of the test results.

[0081] S2. Determine the illumination source and chip scanning path. Based on the illumination source and chip scanning path, scan multiple solder joint areas in the fixed chip to obtain a set of chip grayscale images, in which the chip grayscale images correspond one-to-one with the solder joint areas.

[0082] It should be explained that the illumination source is an optical light source used to illuminate the solder joints of the chip. For example, the illumination source is a visible light or near-infrared light source. The chip scanning path is a pre-defined route for scanning the chip solder joint array area by area, row by row, or point by point.

[0083] S3. Perform the following operations on each chip grayscale image in the chip grayscale image set.

[0084] It should be explained that a chip grayscale image set is a collection of multiple grayscale images obtained through scanning and imaging. A chip grayscale image is a two-dimensional digital image acquired by an optical imaging device from a single solder joint area on the chip, retaining only brightness information and excluding color information.

[0085] S4. Perform solder joint region segmentation on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image.

[0086] Specifically, the step of segmenting the solder joint region of the chip grayscale image to obtain an optimized binary edge image includes:

[0087] Median filtering is performed on the chip grayscale image to obtain a denoised chip grayscale image. Histogram equalization is then performed on the denoised chip grayscale image to obtain an equalized chip grayscale image.

[0088] Image correction is performed on the equalized chip grayscale image to obtain a corrected chip grayscale image, wherein the corrected chip grayscale image includes multiple grayscale pixels;

[0089] Binary edge images are obtained by acquiring grayscale images from correction chips, and morphological closing operations are performed on the binary edge images to obtain optimized binary edge images.

[0090] It should be explained that the median filtering operation on the chip grayscale image involves using a sliding filter window of a preset size to traverse every pixel in the chip grayscale image. The grayscale values ​​of all pixels within the coverage area of ​​the sliding filter window are sorted by size, and the median value of the sorted pixels replaces the original grayscale value of the current center pixel, thus achieving image denoising. The denoised chip grayscale image is the chip grayscale image after median filtering. The histogram equalization operation on the denoised chip grayscale image involves statistically analyzing the pixel distribution probability of each grayscale level in the denoised chip grayscale image. A grayscale mapping function is used to stretch the grayscale distribution of the original image to the entire grayscale range, enhancing the overall image contrast and highlighting the grayscale difference between the solder joint area and the background area. The equalized chip grayscale image is the image obtained after histogram equalization of the denoised chip grayscale image. The image correction operation on the equalized chip grayscale image is a geometric correction operation to eliminate the influence of imaging distortion on the equalized chip grayscale image. The calibration chip grayscale image is the image obtained after image correction of the equalization chip grayscale image. It should be noted that the median filtering, histogram equalization, and image correction operations described above can all be implemented using existing technologies, and will not be elaborated upon here. A grayscale pixel is the smallest unit constituting the calibration chip grayscale image. Each grayscale pixel has an independent grayscale value, used to represent the brightness information at the corresponding location.

[0091] Understandably, the detailed steps for obtaining the binary edge image based on the grayscale image of the correction chip will be given later. Morphological closing operation is a classic morphological operation in digital image processing and is a mature existing technology, so it will not be elaborated here. The optimized binary edge image is the image obtained after performing morphological closing operation on the binary edge image. The optimized binary edge image has continuous edges, less noise interference, and complete solder joint contours, and can be directly used for subsequent identification and judgment of solder joint defects.

[0092] Specifically, the acquisition of a binary edge image based on the grayscale image of the correction chip includes:

[0093] Gray pixels are extracted sequentially from the grayscale image of the calibration chip. The grayscale pixel values ​​are determined based on the extracted grayscale pixels. The adjacent horizontal grayscale pixel values ​​and adjacent vertical grayscale pixel values ​​are determined from the grayscale image of the calibration chip based on the grayscale pixel values.

[0094] By subtracting the grayscale pixel value from the adjacent horizontal grayscale pixel value and the adjacent vertical grayscale pixel value respectively, we can obtain the gradient value in the horizontal axis direction and the gradient value in the vertical axis direction.

[0095] Calculate the pixel gradient magnitude based on the gradient values ​​in the horizontal and vertical directions, obtain the gradient threshold, and compare the pixel gradient magnitude with the gradient threshold.

[0096] If the pixel gradient magnitude is greater than or equal to the gradient threshold, the extracted grayscale pixel is taken as the edge pixel of the solder joint.

[0097] If the pixel gradient magnitude is less than the gradient threshold, the extracted grayscale pixel will be used as the background pixel.

[0098] The solder joint edge pixels and background pixels are summarized separately to obtain the solder joint edge pixel set and the background pixel set. The binary edge image is then identified based on the solder joint edge pixel set and the background pixel set.

[0099] It should be explained that the grayscale pixel value is the brightness value corresponding to a single grayscale pixel extracted from the grayscale image of the calibration chip. The brightness value ranges from [0, 255]. Adjacent horizontal grayscale pixel values ​​are the grayscale values ​​corresponding to grayscale pixels that are in the same horizontal row (horizontal) as the extracted grayscale pixel and are adjacent to it. Adjacent vertical grayscale pixel values ​​are the grayscale values ​​corresponding to grayscale pixels that are in the same vertical column (vertical) as the extracted grayscale pixel and are adjacent to it. The horizontal and vertical gradient values ​​are the pixel values ​​obtained by subtracting the grayscale pixel value from the adjacent horizontal grayscale pixel value and the pixel value obtained by subtracting the grayscale pixel value from the adjacent vertical grayscale pixel value, respectively. The pixel gradient magnitude is the value obtained by taking the square root of the square of the horizontal gradient value plus the square of the vertical gradient value. The gradient threshold is a critical value used to distinguish solder joint edge pixels from background pixels. The gradient threshold is obtained by: obtaining the mean and standard deviation of all grayscale pixels in the grayscale image of the calibration chip, and then multiplying the mean and standard deviation by a preset adjustment coefficient (such as 1) to obtain the value, which is the gradient threshold.

[0100] For example, the grayscale image of the correction chip is Then the grayscale pixel value is The adjacent horizontal grayscale pixel values ​​are The adjacent vertical grayscale pixel values ​​are It should be noted that this invention employs the forward difference method in gradient calculation. This method, when calculating the horizontal gradient, defaults to increasing the row index of the current pixel as the position of the adjacent horizontal grayscale pixel value, and increasing the column index of the current pixel as the position of the adjacent vertical grayscale pixel value. The purpose is to detect the changing trend of grayscale values ​​as the coordinates increase, effectively capturing the abrupt grayscale change from the background (dark) to the solder joint (bright) at the solder joint edge. Therefore, in the example above, the adjacent horizontal grayscale pixel value of 50 is 200, so it is taken as 200. The other adjacent pixel, 10, although also an adjacent pixel, is not considered in the forward difference method. Similarly, the vertical gradient is taken from the adjacent pixel below, 255, also based on the same forward difference principle.

[0101] Importantly, if the pixel gradient magnitude is greater than or equal to the gradient threshold, it indicates a drastic grayscale change at that grayscale pixel, consistent with the grayscale characteristics of solder joint edge regions (significant grayscale differences between the solder joint edge and the background, or within the solder joint itself). Therefore, this pixel can be identified as a solder joint edge pixel. Solder joint edge pixels are grayscale pixels with a pixel gradient magnitude greater than or equal to the gradient threshold. If the pixel gradient magnitude is less than the gradient threshold, it indicates a smooth grayscale change at that grayscale pixel, without significant abrupt grayscale changes, consistent with the grayscale characteristics of non-edge regions of the solder joint (such as within the solder joint or the chip background area). Therefore, this pixel can be identified as a background pixel. Background pixels are grayscale pixels with a pixel gradient magnitude less than the gradient threshold. The solder joint edge pixel set is a collection of solder joint edge pixels. The background pixel set is a collection of background pixels. The steps for identifying a binary edge image based on the solder joint edge pixel set and the background pixel set are as follows: assign a pixel value (e.g., 255, which appears white) to the pixels corresponding to the solder joint edge pixel set, and assign another pixel value (e.g., 0, which appears black) to the pixels corresponding to the background pixel set, thus forming a binary image with clear solder joint edges and obvious contrast between the background and the edges.

[0102] S5. Create a blank mask, and fill the solder joint contour on the blank mask based on the optimized binary edge image to obtain the solder joint mask image.

[0103] It should be explained that the blank mask is an image with the same size as the optimized binary edge image, and all pixels are initially set to background pixel values ​​(e.g., 0, all black). The process of filling the solder joint contour on the blank mask based on the optimized binary edge image is as follows: using the solder joint edge pixels in the optimized binary edge image as the contour boundary, the internal region enclosed by the solder joint edge pixels in the blank mask is filled with pixel values, and the internal region is uniformly set to the solder joint region pixel value (e.g., 255), so that the blank mask forms a solid region consistent with the shape and position of the solder joint, thus completing the solder joint contour filling. The solder joint mask image is a binary image obtained by filling the solder joint contour based on the blank mask according to the optimized binary edge image.

[0104] S6. Based on the solder joint mask image, perform image feature extraction on the grayscale image of the calibration chip to obtain the feature vector.

[0105] In detail, the image feature extraction operation based on the solder joint mask image to the grayscale image of the calibration chip to obtain a feature vector includes:

[0106] Gray pixels are extracted sequentially from the grayscale image of the calibration chip. The extracted gray pixels are used as gray pixels to be calculated. The target mask value is determined from the solder joint mask image based on the gray pixels to be calculated.

[0107] If the target mask value is equal to the preset black pixel value, then the target mask value is used to replace the pixel value of the grayscale pixel to be processed, and the updated grayscale pixel is obtained.

[0108] If the target mask value is not equal to the black pixel value, then the grayscale pixel to be calculated is used as the solder joint grayscale pixel;

[0109] The updated grayscale pixels and solder joint grayscale pixels are summarized separately to obtain the updated grayscale pixel set and the solder joint grayscale pixel set. The image to be detected is identified based on the updated grayscale pixel set and the solder joint grayscale pixel set.

[0110] Geometric and grayscale feature parameters are calculated based on the image to be detected. The geometric and grayscale feature parameters are then integrated to obtain a feature vector.

[0111] It should be explained that the grayscale pixel to be processed is a single grayscale pixel extracted from the grayscale image of the calibration chip. The target mask value is a pixel value extracted from the solder joint mask image that corresponds perfectly in spatial location to the grayscale pixel to be processed. The black pixel value is a pre-set pixel value representing the background area in the solder joint mask image. If the target mask value is equal to the preset black pixel value, it means that the mask area corresponding to the grayscale pixel to be processed is a background area (non-solder joint area), and the grayscale pixel to be processed belongs to the grayscale pixel corresponding to the background. It needs to be replaced by pixel value to mask its interference with subsequent feature extraction, ensuring that only the grayscale information of the solder joint area is retained. It should be noted that the black pixel value is 0. Updating the grayscale pixel is to replace the original grayscale value of the grayscale pixel to be processed with the target mask value to obtain a new pixel. If the target mask value is not equal to the black pixel value, it means that the mask area corresponding to the grayscale pixel to be processed is a solder joint area, and the grayscale pixel to be processed belongs to the grayscale pixel corresponding to the solder joint. No pixel value replacement is required. The solder joint grayscale pixels are grayscale pixels whose target mask values ​​are not equal to the black pixel values ​​to be processed. The updated grayscale pixel set is a set composed of updated grayscale pixels. The solder joint grayscale pixel set is a set composed of solder joint grayscale pixels. The image to be detected is an image composed of the updated grayscale pixel set and the solder joint grayscale pixel set. The detailed steps for calculating geometric feature parameters and grayscale feature parameters based on the image to be detected will be given later. The integration of geometric feature parameters and grayscale feature parameters is the operation of placing geometric feature parameters and grayscale feature parameters in the same set. The feature vector is a sequence obtained by integrating the geometric feature parameters and grayscale feature parameters of the image to be detected.

[0112] It should be noted that in the above steps of this invention, the solder joint mask image can clearly distinguish between the solder joint area and the background area. The target mask value is used to determine whether the grayscale pixel to be processed belongs to the solder joint area, thereby accurately filtering out solder joint-related pixels and avoiding interference from background pixels in subsequent feature extraction. When the target mask value is equal to a preset black pixel value, it indicates that the grayscale pixel to be processed corresponds to the background area. The updated grayscale pixel is obtained by replacing the original pixel value with the target mask value, and the background area can be uniformly set to the background value, thus completely shielding background interference and ensuring that feature extraction is performed only on the solder joint area. When the target mask value is not equal to the black pixel value, it indicates that the grayscale pixel to be processed corresponds to the solder joint area. Its original grayscale information is directly retained as the solder joint grayscale pixel, which can completely preserve the true grayscale features of the solder joint and provide reliable data for subsequent feature calculation. Subsequently, the image to be detected is reconstructed based on the updated grayscale pixel set and the solder joint grayscale pixel set, resulting in an image with a clean background and prominent solder joint information. Finally, geometric feature parameters and grayscale feature parameters are calculated and integrated based on the image to be detected to form a feature vector. The geometric feature parameters can characterize the shape of the solder joint, and the grayscale feature parameters can reflect the grayscale distribution of the solder joint. The combination of the two can comprehensively characterize the characteristics of the solder joint and convert the shape and grayscale information of the solder joint into a standardized digital sequence, which is convenient for subsequent defect recognition model calls and defect judgment.

[0113] Specifically, the calculation of geometric feature parameters and grayscale feature parameters based on the image to be detected includes:

[0114] Based on the image to be detected, the region of the solder joint to be tested and the set of image pixel values ​​are obtained. The minimum bounding rectangle and the outline of the solder joint region are obtained based on the region of the solder joint to be tested. The area of ​​the solder joint is calculated based on the region of the solder joint to be tested.

[0115] Obtain the rectangle width and matrix height based on the minimum bounding rectangle, and calculate the aspect ratio based on the rectangle width and matrix height;

[0116] The perimeter of the outline is calculated based on the weld point area, and the roundness is calculated based on the perimeter of the outline and the area of ​​the weld point.

[0117] The mean value of the solder joint area is calculated based on the image pixel value set, and the gray value variance of the solder joint area is calculated based on the mean value of the solder joint area.

[0118] The maximum and minimum gray values ​​are determined based on the image pixel value set. The difference between the maximum and minimum gray values ​​is calculated to obtain the gray peak value.

[0119] The number of energy pixels is counted based on the image pixel value set and the preset pixel threshold. The energy is calculated based on the number of energy pixels and the solder joint area. The entropy value is calculated based on the energy.

[0120] Geometric feature parameters were determined based on the solder joint area, aspect ratio, and roundness. Gray-scale feature parameters were determined based on the mean gray-scale value, variance gray-scale value, peak-to-peak gray-scale value, energy, and entropy value of the solder joint area.

[0121] It should be explained that the area of ​​the solder joint to be tested is the effective detection area in the image to be detected after being located by the solder joint mask, retaining only the solder joint and shielding the background. The image pixel value set is the set of gray values ​​of all gray-level pixels in the area of ​​the solder joint to be tested. The steps of obtaining the minimum bounding rectangle and the outline of the solder joint area based on the area of ​​the solder joint to be tested are as follows: traverse the boundary pixels of the area of ​​the solder joint to be tested using an edge detection algorithm or a contour extraction algorithm, extract the set of continuous pixels that constitute the outer boundary of the solder joint, forming the outline of the solder joint area. Then, using this outline of the solder joint area as a reference, use the rotating caliper method or the minimum bounding algorithm to traverse and calculate the smallest rectangle that can completely enclose the outline, thus obtaining the minimum bounding rectangle of the area of ​​the solder joint to be tested. The area of ​​the solder joint is the total number of pixels contained in the area of ​​the solder joint to be tested. The width of the rectangle is the side length of the minimum bounding rectangle in the horizontal direction. The height of the matrix is ​​the side length of the minimum bounding rectangle perpendicular to the horizontal direction. The steps of calculating the aspect ratio based on the rectangle width and the matrix height are as follows: take the maximum and minimum values ​​of the rectangle width and the matrix height, and divide the maximum value by the minimum value to obtain the aspect ratio. The perimeter of the outline is the total length obtained by summing the distances between adjacent pixels on the outline of the solder joint area. Detailed steps for calculating roundness based on the perimeter and solder joint area will be provided later.

[0122] Understandably, the grayscale mean of the solder joint area is the average of all grayscale values ​​in the image pixel set. The grayscale variance of the solder joint area is the average of the squared deviations of each image pixel value from the grayscale mean. The maximum and minimum grayscale values ​​are the largest and smallest grayscale values ​​in the image pixel set, respectively. The peak-to-peak grayscale value is the grayscale value obtained by subtracting the minimum grayscale value from the maximum grayscale value. The peak-to-peak grayscale value reflects the fluctuation range of brightness in the solder joint area under test. The pixel threshold is a pre-set grayscale critical value used to distinguish valid grayscale pixels within the solder joint area. The pixel threshold is set as follows: A large number of solder joint area samples are collected, and grayscale histograms are plotted for each solder joint area sample. Based on the grayscale histograms, the grayscale peaks corresponding to the dark gray areas of normal solder joints and the bright spot areas of defective solder joints are distinguished and determined. The grayscale value corresponding to the valley between the two peaks is selected as the initial pixel threshold. The initial pixel thresholds corresponding to all samples are summarized, and their average value is calculated. This average value is used as the final pixel threshold. The step of counting the number of energy pixels based on the image pixel value set and the preset pixel threshold is as follows: Count the number of image pixel values ​​in the image pixel value set that are greater than the pixel threshold, and use the counted number as the number of energy pixels. Energy is the value obtained by dividing the number of energy pixels by the area of ​​the solder joint. The formula for calculating the entropy value based on energy is as follows:

[0123] ;

[0124] in, Represents the entropy value. Indicates pixel threshold, Indicates energy. This represents a logarithmic function. Geometric characteristic parameters are composed of the solder joint area, aspect ratio, and roundness. Gray-level characteristic parameters are composed of the mean gray level of the solder joint area, the variance of the gray level of the solder joint area, the peak-to-peak gray level, energy, and entropy.

[0125] In detail, the formula for calculating the roundness is as follows:

[0126] ;

[0127] in, Indicates roundness. Indicates the perimeter of the outline. Represents pi (π). This indicates the area of ​​the solder joint.

[0128] It should be explained that roundness is a geometric feature parameter characterizing the regularity and near-circular shape of a solder joint outline. In the formula of this invention, the ratio of the square of the outline perimeter to 4π times the solder joint area characterizes the degree to which the solder joint outline approximates a standard circle, reflecting the regularity and smoothness of the solder joint's shape. When the solder joint outline is close to a standard circle, the ratio of its outline perimeter to the solder joint area conforms to the characteristics of a standard circle, and the roundness value is close to 1. When the solder joint has defects (such as deformation, missing corners, irregular protrusions, bridging, etc.), its outline perimeter will increase relatively, while the solder joint area will change less, causing the roundness value to deviate from 1, and the more severe the defect, the greater the deviation.

[0129] S7. Summarize the feature vectors to obtain the feature vector set corresponding to the chip grayscale image set.

[0130] It should be explained that the eigenvector set is a collection of eigenvectors.

[0131] S8. Normalize the feature vector set to obtain the normalized feature vector set, construct the solder joint defect identification model, and input the normalized feature vector set into the solder joint defect identification model to obtain the normal solder joint set and the defective solder joint set.

[0132] Specifically, the construction of the solder joint defect identification model includes:

[0133] Obtain the training and testing sample sets, and set the BP network weight set and error precision;

[0134] The training sample set and the BP network weight set are both input into the pre-constructed BP network input layer to obtain the network output value set, which includes multiple network output values ​​and corresponds one-to-one with the training samples.

[0135] A set of expected values ​​for samples is obtained based on the training sample set, wherein each expected value corresponds one-to-one with a training sample.

[0136] The error index function value is calculated based on the sample expected value set and the network output value set.

[0137] If the error index function value is not less than the error accuracy, then calculate the Jacobian matrix and the error vector, and calculate the weight increment based on the Jacobian matrix and the error vector;

[0138] The weights of each BP network in the BP network weight set are updated using weight increments to obtain an updated BP network weight set. The updated BP network weight set is then used as the BP network weight set, and the process returns to the step of inputting both the training sample set and the BP network weight set into the pre-constructed BP network input layer.

[0139] If the error index function value is less than the error precision, then the updated BP network weight set is taken as the optimal BP network weight set.

[0140] The LM-BP network was identified based on the optimal BP network weight set. The LM-BP network was then tested using a test sample set to obtain a solder joint defect identification model.

[0141] It should be explained that BP network weights are the weight parameters of the connections between neurons in a BP (Backpropagation) neural network. A BP neural network is a multi-layer feedforward neural network trained using an error backpropagation algorithm. The BP network weight set is a collection of BP network weights. Error precision is the critical error value used to determine whether the BP network training has met requirements. The BP network input layer is the first layer of the BP neural network, used to receive input data such as the training sample set and the BP network weight set, and to pass the data to the hidden layers for feature mapping and computation. The network output value set is the set of results obtained from the output layer after the training sample set and BP network weights are input into the BP network input layer and processed by the network. The network output value is the predicted defect category of a single training sample by the BP neural network. The sample expected value set is the set of expected output values ​​set based on the actual category (normal or defective) of each sample in the training sample set. For example, the sample expected value for a normal solder joint is 1, and the sample expected value for a defective solder joint is 0. The detailed steps for calculating the error index function value based on the sample expected value set and the network output value set will be given later. If the error index function value is not less than the error precision, it indicates that the BP network weights of the current BP neural network have not yet reached the error precision, and the deviation between the network output value and the expected value of the sample exceeds the acceptable range. It is necessary to update the BP network weights by calculating the weight increment and continue iterative training to reduce the error. The steps for calculating the Jacobian matrix and error vector are as follows: The Jacobian matrix is ​​constructed by taking the first-order partial derivatives of the error index function with respect to each weight of the BP network. Each row in the Jacobian matrix corresponds to a training sample, and each column corresponds to a network weight. Each element in the Jacobian matrix represents the degree of influence of a small change in a single BP network weight on the overall error index function under the current BP network weight conditions. This is used to quantitatively describe the relationship between the BP network weights and the error, providing a gradient direction basis for subsequent weight optimization. The error vector is obtained by subtracting the true expected value of each sample in the expected value set from the network output value obtained by forward propagation of the corresponding training sample. Each element in the error vector represents the magnitude and direction of the deviation between the current network's output result for a single training sample and the true expected value, reflecting the overall deviation between the network output and the true label. The steps for calculating the Jacobian matrix and error vector in this invention can all be implemented using existing technologies, and will not be described in detail here.

[0142] Importantly, the formula for calculating the weight increment in the step of calculating the weight increment based on the Jacobian matrix and the error vector is as follows:

[0143] ;

[0144] in, Indicates the weight increment. Represents the Jacobian matrix. Indicates the transpose symbol. This represents the preset learning rate. Represents the identity matrix. This represents the error vector. The learning rate is a pre-set adjustment coefficient during the BP network weight update process, used to control the magnitude of the weight increment and the step size of the network iteration. The formula for calculating the weight increment involves transposing the Jacobian matrix, multiplying the transposed Jacobian matrix by itself, adding the product of the preset learning rate and the identity matrix (α×Q), inverting the resulting matrix, and finally multiplying the inverse matrix by the transposed Jacobian matrix and the error vector sequentially to obtain the weight increment. The purpose is to provide a precise adjustment basis for the iterative update of the BP network weights. By combining the Jacobian matrix (reflecting the degree of influence of the weights on the error) and the error vector (reflecting the deviation between the network output and the true value), and adjusting the step size and stability of the BP network weight update through the learning rate and the identity matrix, it avoids excessively fast BP network weight updates leading to network oscillations or excessively slow updates leading to low training efficiency, ensuring that the BP network weight updates accurately target the direction of error reduction. This formula is used to quantify the magnitude and direction of the adjustment required for the current BP network weights. The magnitude of the weight increment corresponds to the magnitude of the BP network weight adjustment. Its matrix dimension is consistent with that of the BP network weights. Each element corresponds to a specific adjustment amount for a BP network weight. After updating the BP network weights through this weight increment, the error index function value can be effectively reduced, making the network output closer to the expected value and gradually improving the model's learning and recognition accuracy of solder joint defect features.

[0145] It should be noted that the step of updating the BP network weight set using weight increments to obtain the updated BP network weight set is as follows: the weight increments are added to each BP network weight in the BP network weight set to obtain the updated BP network weight set. If the error index function value is less than the error precision, it means that the deviation between the current BP network output value and the expected value of the sample has reached the error precision, the network weight optimization is complete, and no further iterative training is needed. The optimal BP network weights are the BP network weights that minimize the network output error when the error index function value is less than the error precision. The optimal BP network weight set is a set composed of the optimal BP network weights. The step of identifying the LM-BP network based on the optimal BP network weight set is as follows: the optimal BP network weight set is embedded into the BP neural network, and the improved BP neural network obtained by optimizing the weight update process using the Levenberg-Marquardt (LM) algorithm has the characteristics of fast convergence speed and high fitting accuracy. The solder joint defect identification model is a model that can be directly used for chip solder joint defect detection after the identification accuracy of the LM-BP network is verified by using a test sample set.

[0146] Specifically, obtaining the training sample set and the test sample set includes:

[0147] Obtain a chip sample set, identify the known normal solder joint image set and the known defective solder joint image set based on the chip sample set, and obtain the chip sample feature vector set based on the known normal solder joint image set and the known defective solder joint image set.

[0148] The chip sample feature vector set is divided to obtain the training sample set and the test sample set.

[0149] It should be explained that the chip sample is a real-world chip sample used to construct the solder joint defect identification model. This chip sample contains known normal solder joints and various defective solder joints. The chip sample set is a collection of chip samples. The known normal solder joint image set is a collection of solder joint images selected from the chip sample set and confirmed to be defect-free by manual or high-precision inspection. Each known normal solder joint image in the known normal solder joint image set is labeled with a "normal" category label. The known defective solder joint image set is a collection of solder joint images selected from the chip sample set and confirmed to have defects such as cold solder joints, voids, cracks, and misalignments by manual or high-precision inspection. Each known defective solder joint image in the known defective solder joint image set is labeled with a specific defect category label. The method for obtaining the chip sample feature vector set based on the known normal solder joint image set and the known defective solder joint image set is the same as the method for obtaining the feature vector based on the image to be detected, and will not be repeated here. The chip sample feature vector set is a collection of feature vectors formed by extracting the geometric feature parameters and grayscale feature parameters corresponding to each solder joint image based on the known normal solder joint image set and the known defective solder joint image set, and integrating them. The process of dividing the chip sample feature vector set involves randomly or evenly splitting the chip sample feature vector set into two parts according to a preset ratio (e.g., 7:3, 8:2) or by category. The training sample set is the collection of chip sample feature vectors used to train the BP network and adjust the network weights. The test sample set is the collection of chip sample feature vectors that were not used in training and are only used to verify the recognition accuracy of the trained model.

[0150] In detail, the formula for calculating the error index function value is as follows:

[0151] ;

[0152] in, This represents the value of the error index function. This indicates the number of training samples in the training sample set. Represents the first value in the sample expectation set. Expected value of each sample Indicates the first value in the network output value set. Each network output value, This represents the L2 norm.

[0153] It should be noted that in the above formula of this invention, the error index function is... As the calculation coefficients, each training sample in the training sample set is processed one by one, first calculating the first... The error index function is calculated by summing the magnitudes of the differences between the expected values ​​of all training samples and the network output values. This formula quantifies and statistically analyzes the deviation between the predicted outputs and true labels of all training samples under the current weights of the BP neural network. It comprehensively reflects the cumulative deviation between the overall network output and the true expected values ​​of the samples, thus providing a unified, calculable, and comparable error evaluation basis for the iterative training of the BP neural network. This error index function value allows for a direct assessment of whether the current network output meets the error accuracy requirements, thereby determining whether to continue iteratively updating the network weights. Furthermore, the error index function value characterizes the overall fitting effect and recognition accuracy of the BP neural network on the solder joint feature samples. The smaller the error index function value, the smaller the gap between the network output value and the expected value of the samples, indicating that the BP neural network has learned the features of normal and defective solder joints more fully and the mapping relationship is more accurate, thus enabling the constructed solder joint defect recognition model to have higher recognition reliability and stability.

[0154] S9. Perform chip solder joint defect detection based on optical characteristics, using normal solder joint sets and defective solder joint sets.

[0155] It should be explained that the present invention can quickly distinguish and output the corresponding set of normal solder joints and set of defective solder joints, which can get rid of the problems of strong subjectivity, low efficiency and easy omission and false detection in traditional manual inspection, realize the automated, intelligent and standardized identification of solder joint defects, and improve the detection efficiency and identification accuracy.

[0156] To address the problems described in the background art, this invention acquires a flip chip under test, identifies the chip solder joint array surface based on the flip chip, and fixes the flip chip under test based on the chip solder joint array surface to obtain a fixed chip. The fixed chip includes multiple solder joint areas. By fixing the flip chip under test, this invention ensures that the chip maintains a stable position during the testing process, avoiding image acquisition deviations caused by displacement. The invention also determines the illumination light source and chip scanning path, and scans multiple solder joint areas in the fixed chip based on the illumination light source and chip scanning path to obtain a chip grayscale image set. Each chip grayscale image corresponds one-to-one with a solder joint area. This invention ensures consistent illumination conditions and scanning methods, giving the acquired chip grayscale image set a unified imaging standard and improving the reliability and consistency of image data. For each chip grayscale image in the chip grayscale image set, the following operation is performed: solder joint area segmentation is performed on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image. This invention, by optimizing the binary edge image, can accurately extract the solder joint contour and remove background and noise interference, highlighting the effective area of ​​the solder joint. This invention creates a blank mask and fills the solder joint contours of the blank mask based on an optimized binary edge image to obtain a solder joint mask image. This allows for accurate differentiation between the solder joint area and the background area, effectively shielding the influence of irrelevant areas on subsequent processing. Based on the solder joint mask image, image feature extraction is performed on the corrected chip grayscale image to obtain feature vectors. This invention can completely preserve the geometric and grayscale features of the solder joints while removing background interference, improving the accuracy and specificity of feature extraction. The feature vectors are summarized to obtain a feature vector set corresponding to the chip grayscale image set. The feature vector set is normalized to obtain a normalized feature vector set. A solder joint defect identification model is constructed, and the normalized feature vector set is input into the solder joint defect identification model to obtain a set of normal solder joints and a set of defective solder joints. This invention summarizes and normalizes the extracted feature vectors, eliminating dimensional differences and numerical deviations between different features, improving data uniformity and model input stability. Based on the set of normal solder joints and the set of defective solder joints, chip solder joint defect detection based on optical characteristics is completed. Therefore, the present invention can improve the accuracy and efficiency of chip solder joint defect detection.

[0157] like Figure 2 The diagram shown is a functional block diagram of a chip solder joint defect detection system based on optical properties provided in an embodiment of the present invention.

[0158] The optical characteristic-based chip solder joint defect detection system 100 of the present invention can be installed in an electronic device. Depending on the functions implemented, the optical characteristic-based chip solder joint defect detection system 100 may include a chip fixing module 101, an optical image acquisition module 102, an image feature extraction module 103, and a chip defect identification module 104. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0159] The chip fixing module 101 is used to acquire the flip chip to be tested, identify the chip solder joint array surface based on the flip chip to be tested, and fix the flip chip to be tested based on the chip solder joint array surface to obtain a fixed chip, wherein the fixed chip includes multiple solder joint areas.

[0160] The optical image acquisition module 102 is used to determine the illumination light source and the chip scanning path, and scan multiple solder joint areas in the fixed chip based on the illumination light source and the chip scanning path to obtain a chip grayscale image set, wherein the chip grayscale image corresponds one-to-one with the solder joint area.

[0161] The image feature extraction module 103 is used to perform the following operations on each chip grayscale image in the chip grayscale image set: perform solder joint region segmentation on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image, create a blank mask, fill the solder joint contour on the blank mask based on the optimized binary edge image to obtain a solder joint mask image, and perform image feature extraction on the corrected chip grayscale image based on the solder joint mask image to obtain a feature vector;

[0162] The chip defect identification module 104 is used to summarize feature vectors to obtain a feature vector set corresponding to the chip grayscale image set, normalize the feature vector set to obtain a normalized feature vector set, construct a solder joint defect identification model, input the normalized feature vector set into the solder joint defect identification model to obtain a normal solder joint set and a defective solder joint set, and complete the chip solder joint defect detection based on optical characteristics based on the normal solder joint set and the defective solder joint set. Specifically, in this embodiment of the invention, each module in the chip solder joint defect detection system 100 based on optical characteristics adopts the same approach as described above. Figure 1 The method used is the same as the optical property-based chip solder joint defect detection method described in the article, and can produce the same technical effect, so it will not be repeated here.

[0163] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a chip solder joint defect detection method based on optical properties, according to an embodiment of the present invention.

[0164] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a chip solder joint defect detection method program based on optical properties.

[0165] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a chip solder joint defect detection method program based on optical characteristics, but also to temporarily store data that has been output or will be output.

[0166] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a chip solder joint defect detection method program based on optical characteristics) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0167] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0168] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0169] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0170] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0171] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0172] The chip solder joint defect detection method program based on optical characteristics stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0173] Obtain the flip chip to be tested, identify the chip solder joint array surface based on the flip chip to be tested, and fix the flip chip to be tested based on the chip solder joint array surface to obtain a fixed chip, wherein the fixed chip includes multiple solder joint areas;

[0174] The illumination source and chip scanning path are determined. Based on the illumination source and chip scanning path, multiple solder joint areas in the fixed chip are scanned to obtain a set of chip grayscale images, in which the chip grayscale images correspond one-to-one with the solder joint areas.

[0175] Perform the following operations on each chip grayscale image in the chip grayscale image set:

[0176] Solder joint region segmentation is performed on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image;

[0177] Create a blank mask, and fill the solder joint contours on the blank mask based on the optimized binary edge image to obtain the solder joint mask image;

[0178] Image feature extraction is performed on the grayscale image of the calibration chip based on the solder joint mask image to obtain the feature vector;

[0179] By summarizing the feature vectors, we obtain the feature vector set corresponding to the chip grayscale image set;

[0180] The feature vector set is normalized to obtain the normalized feature vector set. A solder joint defect identification model is constructed by inputting the normalized feature vector set into the solder joint defect identification model to obtain the normal solder joint set and the defective solder joint set.

[0181] Based on normal solder joint sets and defective solder joint sets, chip solder joint defect detection is performed based on optical characteristics.

[0182] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0183] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0184] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0185] Obtain the flip chip to be tested, identify the chip solder joint array surface based on the flip chip to be tested, and fix the flip chip to be tested based on the chip solder joint array surface to obtain a fixed chip, wherein the fixed chip includes multiple solder joint areas;

[0186] The illumination source and chip scanning path are determined. Based on the illumination source and chip scanning path, multiple solder joint areas in the fixed chip are scanned to obtain a set of chip grayscale images, in which the chip grayscale images correspond one-to-one with the solder joint areas.

[0187] Perform the following operations on each chip grayscale image in the chip grayscale image set:

[0188] Solder joint region segmentation is performed on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image;

[0189] Create a blank mask, and fill the solder joint contours on the blank mask based on the optimized binary edge image to obtain the solder joint mask image;

[0190] Image feature extraction is performed on the grayscale image of the calibration chip based on the solder joint mask image to obtain the feature vector;

[0191] By summarizing the feature vectors, we obtain the feature vector set corresponding to the chip grayscale image set;

[0192] The feature vector set is normalized to obtain the normalized feature vector set. A solder joint defect identification model is constructed by inputting the normalized feature vector set into the solder joint defect identification model to obtain the normal solder joint set and the defective solder joint set.

[0193] Based on normal solder joint sets and defective solder joint sets, chip solder joint defect detection is performed based on optical characteristics.

[0194] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0195] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0197] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A chip solder joint defect detection method based on optical characteristics, characterized by, The method includes: Obtain the flip chip to be tested, identify the chip solder joint array surface based on the flip chip to be tested, and fix the flip chip to be tested based on the chip solder joint array surface to obtain a fixed chip, wherein the fixed chip includes multiple solder joint areas; The illumination source and chip scanning path are determined. Based on the illumination source and chip scanning path, multiple solder joint areas in the fixed chip are scanned to obtain a set of chip grayscale images, in which the chip grayscale images correspond one-to-one with the solder joint areas. Perform the following operations on each chip grayscale image in the chip grayscale image set: Solder joint region segmentation is performed on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image; Create a blank mask, and fill the solder joint contours on the blank mask based on the optimized binary edge image to obtain the solder joint mask image; Image feature extraction is performed on the grayscale image of the calibration chip based on the solder joint mask image to obtain the feature vector; By summarizing the feature vectors, we obtain the feature vector set corresponding to the chip grayscale image set; The feature vector set is normalized to obtain the normalized feature vector set. A solder joint defect identification model is constructed by inputting the normalized feature vector set into the solder joint defect identification model to obtain the normal solder joint set and the defective solder joint set. Based on normal solder joint sets and defective solder joint sets, chip solder joint defect detection is performed based on optical characteristics.

2. The chip solder joint defect detection method based on optical properties as described in claim 1, characterized in that, The step of segmenting the solder joint region of the chip grayscale image to obtain an optimized binary edge image includes: Median filtering is performed on the chip grayscale image to obtain a denoised chip grayscale image. Histogram equalization is then performed on the denoised chip grayscale image to obtain an equalized chip grayscale image. Image correction is performed on the equalized chip grayscale image to obtain a corrected chip grayscale image, wherein the corrected chip grayscale image includes multiple grayscale pixels; Binary edge images are obtained by acquiring grayscale images from correction chips, and morphological closing operations are performed on the binary edge images to obtain optimized binary edge images.

3. The chip solder joint defect detection method based on optical properties as described in claim 2, characterized in that, The method of obtaining a binary edge image based on the grayscale image of the correction chip includes: Gray pixels are extracted sequentially from the grayscale image of the calibration chip. The grayscale pixel values ​​are determined based on the extracted grayscale pixels. The adjacent horizontal grayscale pixel values ​​and adjacent vertical grayscale pixel values ​​are determined from the grayscale image of the calibration chip based on the grayscale pixel values. By subtracting the grayscale pixel value from the adjacent horizontal grayscale pixel value and the adjacent vertical grayscale pixel value respectively, we can obtain the gradient value in the horizontal axis direction and the gradient value in the vertical axis direction. Calculate the pixel gradient magnitude based on the gradient values ​​in the horizontal and vertical directions, obtain the gradient threshold, and compare the pixel gradient magnitude with the gradient threshold. If the pixel gradient magnitude is greater than or equal to the gradient threshold, the extracted grayscale pixel is taken as the edge pixel of the solder joint. If the pixel gradient magnitude is less than the gradient threshold, the extracted grayscale pixel will be used as the background pixel. The solder joint edge pixels and background pixels are summarized separately to obtain the solder joint edge pixel set and the background pixel set. The binary edge image is then identified based on the solder joint edge pixel set and the background pixel set.

4. The chip solder joint defect detection method based on optical properties as described in claim 3, characterized in that, The image feature extraction operation based on the solder joint mask image to the grayscale image of the correction chip is used to obtain a feature vector, including: Gray pixels are extracted sequentially from the grayscale image of the calibration chip. The extracted gray pixels are used as gray pixels to be calculated. The target mask value is determined from the solder joint mask image based on the gray pixels to be calculated. If the target mask value is equal to the preset black pixel value, then the target mask value is used to replace the pixel value of the grayscale pixel to be processed, and the updated grayscale pixel is obtained. If the target mask value is not equal to the black pixel value, then the grayscale pixel to be calculated is used as the solder joint grayscale pixel; The updated grayscale pixels and solder joint grayscale pixels are summarized separately to obtain the updated grayscale pixel set and the solder joint grayscale pixel set. The image to be detected is identified based on the updated grayscale pixel set and the solder joint grayscale pixel set. Geometric and grayscale feature parameters are calculated based on the image to be detected. The geometric and grayscale feature parameters are then integrated to obtain a feature vector.

5. The chip solder joint defect detection method based on optical properties as described in claim 4, characterized in that, The calculation of geometric feature parameters and grayscale feature parameters based on the image to be detected includes: Based on the image to be detected, the region of the solder joint to be tested and the set of image pixel values ​​are obtained. The minimum bounding rectangle and the outline of the solder joint region are obtained based on the region of the solder joint to be tested. The area of ​​the solder joint is calculated based on the region of the solder joint to be tested. Obtain the rectangle width and matrix height based on the minimum bounding rectangle, and calculate the aspect ratio based on the rectangle width and matrix height; The perimeter of the outline is calculated based on the weld point area, and the roundness is calculated based on the perimeter of the outline and the area of ​​the weld point. The mean value of the solder joint area is calculated based on the image pixel value set, and the gray value variance of the solder joint area is calculated based on the mean value of the solder joint area. The maximum and minimum gray values ​​are determined based on the image pixel value set. The difference between the maximum and minimum gray values ​​is calculated to obtain the gray peak value. The number of energy pixels is counted based on the image pixel value set and the preset pixel threshold. The energy is calculated based on the number of energy pixels and the solder joint area. The entropy value is calculated based on the energy. Geometric feature parameters were determined based on the solder joint area, aspect ratio, and roundness. Gray-scale feature parameters were determined based on the mean gray-scale value, variance gray-scale value, peak-to-peak gray-scale value, energy, and entropy value of the solder joint area.

6. The chip solder joint defect detection method based on optical properties as described in claim 5, characterized in that, The formula for calculating the roundness is as follows: ； in, Indicates roundness. Indicates the perimeter of the outline. Represents pi (π). This indicates the area of ​​the solder joint.

7. The chip solder joint defect detection method based on optical properties as described in claim 6, characterized in that, The construction of the solder joint defect identification model includes: Obtain the training and testing sample sets, and set the BP network weight set and error precision; The training sample set and the BP network weight set are both input into the pre-constructed BP network input layer to obtain the network output value set, which includes multiple network output values ​​and corresponds one-to-one with the training samples. A set of expected values ​​for samples is obtained based on the training sample set, wherein each expected value corresponds one-to-one with a training sample. The error index function value is calculated based on the sample expected value set and the network output value set. If the error index function value is not less than the error accuracy, then calculate the Jacobian matrix and the error vector, and calculate the weight increment based on the Jacobian matrix and the error vector; The weights of each BP network in the BP network weight set are updated using weight increments to obtain an updated BP network weight set. The updated BP network weight set is then used as the BP network weight set, and the process returns to the step of inputting both the training sample set and the BP network weight set into the pre-constructed BP network input layer. If the error index function value is less than the error precision, then the updated BP network weight set is taken as the optimal BP network weight set. The LM-BP network was identified based on the optimal BP network weight set. The LM-BP network was then tested using a test sample set to obtain a solder joint defect identification model.

8. The chip solder joint defect detection method based on optical properties as described in claim 7, characterized in that, The acquisition of the training sample set and the test sample set includes: Obtain a chip sample set, identify the known normal solder joint image set and the known defective solder joint image set based on the chip sample set, and obtain the chip sample feature vector set based on the known normal solder joint image set and the known defective solder joint image set. The chip sample feature vector set is divided to obtain the training sample set and the test sample set.

9. The chip solder joint defect detection method based on optical properties as described in claim 8, characterized in that, The formula for calculating the error index function value is as follows: ； in, This represents the error index function value. This indicates the number of training samples in the training sample set. Represents the first value in the sample expectation set. Expected value of each sample Indicates the first value in the network output value set. Each network output value, This represents the L2 norm.

10. A chip solder joint defect detection system based on optical properties, characterized in that, The system includes: The chip fixing module is used to acquire the flip chip under test, identify the chip solder joint array surface based on the flip chip under test, and fix the flip chip under test based on the chip solder joint array surface to obtain a fixed chip. The fixed chip includes multiple solder joint areas. The optical image acquisition module is used to determine the illumination light source and the chip scanning path. Based on the illumination light source and the chip scanning path, multiple solder joint areas in the fixed chip are scanned to obtain a set of chip grayscale images, in which the chip grayscale images correspond one-to-one with the solder joint areas. The image feature extraction module performs the following operations on each chip grayscale image in the chip grayscale image set: performs solder joint region segmentation on the chip grayscale image to obtain an optimized binary edge image and a corrected chip grayscale image; creates a blank mask; fills the solder joint contour on the blank mask based on the optimized binary edge image to obtain a solder joint mask image; and performs image feature extraction on the corrected chip grayscale image based on the solder joint mask image to obtain a feature vector. The chip defect identification module is used to summarize feature vectors to obtain the feature vector set corresponding to the chip grayscale image set. The feature vector set is normalized to obtain the normalized feature vector set. A solder joint defect identification model is constructed. The normalized feature vector set is input into the solder joint defect identification model to obtain the normal solder joint set and the defective solder joint set. Based on the normal solder joint set and the defective solder joint set, the chip solder joint defect detection based on optical characteristics is completed.

Images