A solder defect detection method and system based on small target recognition

Abstract

The present invention discloses a solder defect detection method and system based on small target recognition. The method collects solder images, performs image enhancement on the solder images, obtains a pseudo data set of the solder images, constructs a fast deformation convolution kernel, and based on the fast deformation convolution kernel, adopts a YOLOv5 neural network to construct a solder defect detection network and adopts a pseudo data set of the solder images to train the solder defect detection network. The trained solder defect detection network is used to identify solder defects, thereby solving the technical problem of low defect detection accuracy of micron-level solders. Solder defects can be identified quickly and accurately through the constructed solder defect detection network.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of eutectic patch technology, and in particular to a solder defect detection method and system based on small target recognition. Background Art

[0002] During the semiconductor chip production process, the chip must be packaged with a carrier such as a substrate. Eutectic bonding technology is widely used in the electronic packaging industry due to its advantages of high thermal conductivity, high reliability, and low resistance. Eutectic bonding technology involves oxide scrubbing and solder heating and eutectic melting. If the solder has defects such as contamination and unevenness, it will affect the fluidity and thermal conductivity of the eutectic material, thereby causing voids in the eutectic layer and affecting the eutectic quality. To ensure the efficient and stable execution of the eutectic bonding process, solder inspection in the substrate is essential. Optical imaging of the solder surface is a low-cost and efficient detection method. However, solder surface defects are characterized by micron-scale size, irregular shapes, and low background contrast. Therefore, traditional image processing methods cannot accurately distinguish surface defects from normal areas, making them difficult to apply to actual eutectic solder inspection processes. To this end, an image processing method is designed to identify eutectic solder surface defects under non-uniform lighting conditions. This method provides high-precision real-time solder surface information for eutectic bonding technology, which is used to guide subsequent process execution, ensure the stable operation of the eutectic bonding process, and improve the yield rate.

[0003] Currently, solder detection methods can be mainly divided into optical detection and thermodynamic detection.

[0004] Optical inspection methods involve directly capturing images using light reflected from the solder surface, followed by visual inspection or microscopic examination. However, these methods are inefficient and incapable of detecting even tiny substrate solder defects. Machine vision inspection, on the other hand, involves optical imaging, image enhancement using algorithms, and defect identification using image processing or machine learning methods. However, image processing methods rely on manually defined parameters and require fewer steps, resulting in low accuracy and generalizability. Existing machine learning methods fail to effectively address factors such as small defect size and irregular shape, resulting in low recognition accuracy and overlap.

[0005] The thermodynamic method directly heats and cools the solder, analyzing the temperature curve to determine the solder's thermodynamic properties, providing direct information on solder quality. This method offers advantages such as immunity to non-critical defects, high detection accuracy, and the ability to detect internal defects. However, it is typically used for large-volume solder testing. In the eutectic die bonding process, the solder is stored in a solder tank. Separate heating would be costly and difficult to directly measure temperature. Therefore, this method is not suitable for eutectic solder pre-testing.

[0006] The patent application with publication number CN112767345A discloses a eutectic defect detection method based on deep learning. This patent uses a pseudo-sample generation method and image processing technology to generate similar sample data based on existing samples, thereby achieving the goal of expanding the data set. The neural network part of this patent adopts an instance segmentation model based on deep learning, which can obtain the defect location and area based on the trained parameters and input image. Using this method, the impact of the lack of sample sets and the imbalance of positive and negative examples in the training of the eutectic defect detection network can be reduced, and relatively high-precision defect information can be obtained to guide subsequent operations. However, the data expansion part of this method is too random. Although it can significantly increase the number of pseudo-samples, the difference between the pseudo-samples and the real samples is too large, making it difficult to provide effective information. In addition, the neural network structure of this method is not optimized for defect characteristics, information extraction is insufficient, and reliability will be reduced.

[0007] Patent publication number CN108778593A discloses a solder bath impurity diagnosis framework and method based on temperature curves. This patent records the temperature change curve of the solder to be tested and compares it to the reference temperature change curve of normal solder through an algorithm to determine whether the solder to be tested has abnormal thermodynamic properties, thereby determining the overall impurity situation of the solder based on the thermodynamic properties of the solder. This method can effectively detect internal and external impurities and contamination defects in the solder, but it has problems such as low detection efficiency and high detection cost. It is only suitable for testing solder baths and large amounts of solder and cannot detect the flatness of the solder surface. Therefore, it is not suitable for testing eutectic chip solder. Summary of the Invention

[0008] The solder defect detection method and system based on small target recognition provided by the present invention solve the technical problem of low accuracy in defect detection of micron-level solder.

[0009] To solve the above technical problems, the present invention proposes a solder defect detection method based on small target recognition, which includes:

[0010] Collect solder images and perform image enhancement on the solder images.

[0011] Get a fake dataset of solder images.

[0012] A fast deformable convolution kernel is constructed, and based on the fast deformable convolution kernel, a YOLOv5 neural network is used to build a solder defect detection network.

[0013] A pseudo dataset of solder images is used to train a solder defect detection network, and the trained solder defect detection network is used to identify solder defects.

[0014] Furthermore, performing image enhancement on the solder image includes:

[0015] Determine candidate edge regions of the solder image.

[0016] Calculate the edge coefficient of the candidate edge region, which is used to represent the degree of fluctuation of pixels in the candidate edge region.

[0017] According to the edge coefficient, the scale weight of each channel of the solder image is calculated.

[0018] According to the scale weight of each channel of the solder image, a multi-scale illumination enhancement algorithm is used to enhance the solder image.

[0019] Furthermore, the calculation formula of the edge coefficient is:

[0020]

[0021] Among them, β n (x, y) represents the edge coefficient of the candidate edge region with the center coordinate (x, y) in channel n, H n (x,y) represents the candidate edge region of channel n, S(H n (x,y)) represents the candidate edge region H in channel n n The standard deviation of all pixels in (x,y), σ n Indicates the standard deviation of channel n, Max(σ n ) and Min(σ n ) represent σ n The maximum and minimum values ​​of .

[0022] Furthermore, according to the edge coefficient, the calculation formula for the scale weight of each channel of the solder image is:

[0023]

[0024] Among them, ω n (x,y) represents the scale weight of the pixel with coordinates (x,y) in channel n, β n (x,y) represents the edge coefficient of the candidate edge region with the center coordinate (x,y) in channel n, σ n represents the standard deviation of channel n, σ i represents the standard deviation of channel i, N represents the total number of channels of the solder image, and N=3, i and n are both positive integers greater than 0 and less than 4.

[0025] Furthermore, the pseudo dataset of solder images is obtained including:

[0026] Defect annotation is performed on the solder images in the solder image set, and the defect annotation specifically includes annotation of the defect category, coordinates, and length and width values.

[0027] The solder position of the solder image is extracted using Otsu's binarization method, and the chip is straightened through nearest neighbor filling affine transformation to move the solder to the center of the solder image.

[0028] Solder images are divided into normal images and defective images according to whether they have defects, and the K-means clustering method is used on the defective images to obtain the classified solder images.

[0029] The classified solder images are randomly cropped to obtain random cropped areas.

[0030] The randomly cropped regions belonging to the same class in the solder image set are randomly spliced ​​and scaled to obtain a pseudo dataset of solder images.

[0031] Furthermore, the pseudo dataset of solder images is obtained including:

[0032] A normal image and a defective image are randomly selected and transparently merged to obtain a first merged image, wherein the transparency of the normal image and the defective image is a random value and the sum is 1.

[0033] The transparency of the first merged image is calculated, and the first merged image is classified into the same category as the image with low transparency among the normal image and the defective image.

[0034] According to the first merged image determined by the category, a pseudo dataset of solder images is obtained.

[0035] Furthermore, the specific formula for constructing the fast deformable convolution kernel is:

[0036]

[0037] Among them, Y(p0) represents the result of convolution operation on pixel p0 in the solder image, ω n represents the weight of channel n, N represents the total number of solder image channels, and N = 3, ξ n (·) is a single-channel deformable convolution operator for channel n, p ξ The convolution kernel K of the single-channel deformable convolution ξ The position enumerator of the elements, K ξ The set of weights at channel n, R n is the channel n component of the solder image, G is the bilinear interpolation function, X(p0,p ξ ),Y(p0,p ξ ) are based on (p0,p ξ ) The calculated convolution kernel offset in the X and Y directions, p δ is the convolution kernel K of the offset convolution δ The position enumerator of the elements, and K δ In the weight set of channel w and channel h, p0(x) and p0(y) represent the coordinates of p0 in the X and Y directions respectively. ξ (x) and p ξ (y) represents p ξ Coordinates in the X and Y directions.

[0038] The solder defect detection system based on small target recognition provided by the present invention includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the solder defect detection method based on small target recognition provided by the present invention are implemented.

[0039] The solder defect method and system based on small target recognition provided by the present invention collect solder images, perform image enhancement on the solder images, obtain a pseudo data set of the solder images, construct a fast deformation convolution kernel, and based on the fast deformation convolution kernel, use the YOLOv5 neural network to construct a solder defect detection network and use the pseudo data set of the solder images to train the solder defect detection network. The trained solder defect detection network is used to identify solder defects, which solves the technical problem of low defect detection accuracy of micron-level solders. Solder defects can be identified quickly and accurately through the constructed solder defect detection network.

[0040] Specifically, the beneficial effects of the present invention include:

[0041] (1) A small-target-based eutectic solder defect detection method was designed to achieve highly efficient optically stable identification of eutectic solder defects on micron-scale substrates.

[0042] (2) Based on the image prior features, a non-balanced multi-scale illumination enhancement image preprocessing algorithm is proposed, which achieves high-quality optical enhancement of the image while ensuring that the original texture structure of the image is not distorted.

[0043] (3) A fast convolutional target recognition network with deformable convolution kernel was designed to achieve efficient and high-precision solder image target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a data preprocessing diagram of the solder defect detection method based on small target recognition according to the third embodiment of the present invention;

[0045] Figure 2 This is a diagram of a fast deformation convolution structure of a solder defect detection method based on small target recognition according to a third embodiment of the present invention;

[0046] Figure 3 This is a network structure diagram of a solder defect detection method based on small target recognition according to a third embodiment of the present invention;

[0047] Figure 4 This is a flowchart of a neural network training method for solder defect detection based on small target recognition according to the third embodiment of the present invention;

[0048] Figure 5 This is an overall flow chart of a solder defect detection method based on small target recognition according to a third embodiment of the present invention;

[0049] Figure 6 This is a structural block diagram of a solder defect detection system based on small target recognition according to an embodiment of the present invention.

[0050] Reference numerals:

[0051] 10. Memory; 20. Processor. DETAILED DESCRIPTION

[0052] To facilitate understanding of the present invention, the present invention will be described in more comprehensive and detailed form below in conjunction with the accompanying drawings and preferred embodiments. However, the protection scope of the present invention is not limited to the following specific embodiments.

[0053] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, the present invention can be implemented in many different ways as defined and covered by the claims.

[0054] Example 1

[0055] The solder defect detection method based on small target recognition provided in the first embodiment of the present invention includes:

[0056] Step S101: collecting a solder image and performing image enhancement on the solder image.

[0057] Step S102: Acquire a pseudo data set of solder images.

[0058] Step S103: construct a fast deformation convolution kernel, and based on the fast deformation convolution kernel, use the YOLOv5 neural network to construct a solder defect detection network.

[0059] Step S104 : using a pseudo data set of solder images to train a solder defect detection network, and using the trained solder defect detection network to identify solder defects.

[0060] The solder defect detection method based on small target recognition provided by an embodiment of the present invention collects solder images, performs image enhancement on the solder images, obtains a pseudo data set of the solder images, constructs a fast deforming convolution kernel, and based on the fast deforming convolution kernel, adopts a YOLOv5 neural network to construct a solder defect detection network and adopts a pseudo data set of solder images to train the solder defect detection network. The trained solder defect detection network is used to identify solder defects, thereby solving the technical problem of low defect detection accuracy of micron-level solders. Solder defects can be identified quickly and accurately through the constructed solder defect detection network.

[0061] Example 2

[0062] The present invention provides a small-target optimized eutectic solder defect detection method, which solves the problem that micron-level solder defects due to small size, random shape and low contrast are difficult to identify and locate by optical methods.

[0063] To solve these technical problems, the defect detection method proposed in the present invention includes:

[0064] (1) Solder image preprocessing algorithm based on non-balanced multi-scale illumination enhancement to reduce noise caused by factors such as instruments and equipment and image unclearness caused by uneven illumination.

[0065] In order to deal with the impact of uneven illumination on images, an embodiment of the present invention proposes a non-balanced multi-scale illumination enhancement algorithm.

[0066] In Retinex theory, the input image I(x,y) is usually considered to be composed of the illumination image L(x,y) and the reflection image R(x,y). The former refers to the information of the incident light on the object, and the latter represents the reflected part of the object, that is:

[0067] I(x,y)=L(x,y)·R(x,y) (1)

[0068] We cannot obtain the illumination image L(x,y) directly, so we use Gaussian filtering to estimate the original image in the SSR illumination enhancement algorithm:

[0069]

[0070] Transforming Equation (2) into the logarithmic domain and introducing multi-scale illumination enhancement, that is, the weighted average of several single-scale SSRs, the enhanced image can be obtained as follows:

[0071]

[0072] Among them, I n (x,y) represents the pixel value of the nth channel of the input image, R n(x,y) is the output image of the nth channel, N is the number of scales, that is, the number of channels, and R(x,y) is the enhanced image after removing the illumination component. n and ω n (x, y) are the standard deviation of the n-channel Gaussian function and the n-channel scale weight distribution, respectively, which are given by prior knowledge.

[0073] From the above formula, we can get that the SSR algorithm is affected by σ n The impact is greater when σ n When the result is small, the algorithm has a stronger edge preservation effect. n When the convolution value is large, the convolution operation will be affected by a wide range of pixels, thus providing a stronger low-light enhancement effect. The traditional multi-scale illumination enhancement algorithm is manually given by the Gaussian standard deviation σ n and weight distribution ω n (x, y), under different camera internal parameters, lighting environment and image object conditions, manual tuning of hyperparameters is required to improve the actual effect of the algorithm. And the scale weight distribution ω n (x,y) is usually simplified to ω in practical use n , that is, assuming a priori that all pixels in any single channel follow the same distribution, the same weight bias can be used for any point in that channel to achieve optimal enhancement. However, in solder images, the illumination distribution between the solder and the substrate significantly fails to meet this a priori condition. To address this issue and maintain edge contrast while enhancing the image, thereby removing the effects of illumination and maintaining the edge difference between the defect and the solder, an embodiment of the present invention proposes a non-balanced multi-scale illumination enhancement algorithm.

[0074] First, define the single-channel candidate edge region H of the image n (x,y), where n is the corresponding channel, as shown in the following formula:

[0075]

[0076] Among them, I + To fill the image with the nearest neighbor boundary based on I, fill the image with a size of 5, I + (x p ,y p ) is the pixel in the five-neighborhood of the (x, y) nearest neighbor filling image, defined as follows:

[0077]

[0078] W is the image width, H is the image height, close(x p ,y p ) is (x p ,y p)’s nearest neighbor pixel. In order to determine the edge confidence of the candidate edge region of the image, the embodiment of the present invention proposes an edge coefficient β(x p ,y p )Measure area H n The degree of pixel fluctuation of (x,y):

[0079]

[0080] S(H n (x,y)) is the region H n The standard deviation of all pixels in (x, y) can be obtained, so when the area H n The more edges (x,y) contain, the greater the value of β n Therefore, by performing a single-channel candidate edge region traversal query on image I, the edge coefficient β of all pixels in each channel can be obtained. n (x,y) is used to measure the probability that the pixel is an edge pixel.

[0081] In order to make the weight distribution of edge areas as large as possible and the weight distribution of non-edge areas as small as possible, let ω n (x,y) is:

[0082]

[0083] By introducing the above method to measure the edge distribution of the image and constructing the scale weight distribution based on the edge coefficient β, the scale weight distribution is then substituted into formula (3) to achieve image enhancement. This method can achieve low-illumination enhancement of the overall image while maintaining the edge contrast of the image, thereby enhancing the representation of solder defects in the image.

[0084] (2) A solder image augmentation algorithm based on similarity clustering, cropping and filling is proposed to improve the generalization of the network to differential data and avoid model parameter confusion caused by the splicing of similar distribution data.

[0085] In order to solve the problem of small training data sets and imbalanced positive and negative examples, data enhancement methods should be introduced to expand model training samples and improve model detection accuracy and generalization ability. However, the cropping and splicing technology of traditional image enhancement algorithms adopts a random cropping and splicing method. In general data sets, due to the significant differences between different categories, this method can greatly expand the data set, realize the direct extension of the model classification hyperplane, and prompt the decision boundary to move to the low-density area, thereby improving the detection accuracy and generalization of the model. However, in solder defect detection, due to the limitation of data similarity between different categories, the pseudo samples generated after random cropping and classification will be significantly interfered with by different categories, which will bring confusion to the model parameter convergence process and cause the model prediction accuracy to decrease. To solve this problem, the present invention proposes the following data enhancement method:

[0086] Step 1: Collect eutectic solder microscopic images and manually mark defects, marking the defect category, coordinates, and length and width values.

[0087] Step 2: Use Otsu's method to binarize and extract the solder position, and then use the nearest neighbor filling affine transformation to straighten the chip and move the solder to the center of the image.

[0088] Step 3: The collected images are divided into two categories according to whether there are defects or not, and the K-means clustering method is used for the defective images. The K value is 5, so a total of 6 categories are obtained. That is, the image set is traversed and image I is classified into I c , where c is the classification category.

[0089] Step 4: Traverse the image collection and perform c Randomly crop the image into four regions to obtain the image group (w1, w2, h1, h2) are the width and height of the unscaled area, added to the corresponding category set Then from L c Randomly extract four areas, splice them according to the corresponding positions and randomly scale them to obtain pseudo data , as shown below:

[0090]

[0091] in, For L c The results of random extraction and scaling, is the width and height of the area after scaling, The following equations should be satisfied:

[0092]

[0093] Step 5: Randomly extract a normal image and a defective image, and transparently merge them. The transparency of the two images is random and the sum is 1, so as to construct a transparency-merged pseudo image. The label of the pseudo image is the same as the label of the manually annotated image with low transparency. The newly generated image is added to the pseudo dataset.

[0094] Step 6: Extract the defective area from the defective image and randomly overlay it on the same position of the normal image to obtain a randomly overlaid pseudo image, which is then added to the pseudo dataset.

[0095] Through the above data augmentation method, a large number of pseudo data sets can be obtained, and the pseudo data sets have a high degree of similarity with the real data sets, thereby alleviating the problems of missing data sets and imbalance of positive and negative examples.

[0096] (3) A fast deformable convolution method is proposed to decompose the convolution process to reduce computational overhead and improve convergence speed, and the convolution kernel deformation is optimized based on the irregular characteristics of solder defect representation in the input image.

[0097] In actual industrial sites, due to the limitations of the eutectic placement machine's working environment, the workstation will process multiple requests in parallel, such as camera equipment acquisition requests, eutectic welding station real-time curve optimization requests, host computer data requests, working condition storage requests, etc. In order to ensure the life and real-time performance of the equipment, the eutectic placement process places strict requirements on the computational overhead of the defect detection algorithm. The traditional convolution method of exchanging time complexity for network accuracy will cause the time overhead of the detection algorithm to remain high. In traditional convolution, for N k xN k The traditional convolution kernel and input image R, each position p0 in the output feature map Y has:

[0098]

[0099] Where p is the position enumeration of the elements in the convolution kernel K, and Y(p0) is the convolution feature obtained by the convolution kernel with p0 as the center. The above traditional convolution is limited by the shortcomings of the fixed shape of the convolution kernel and the cumbersome calculation process, and the convolution effect is poor for multi-channel complex images. For an image R(W,H,N) with a width of W, a height of H, and a number of channels N, N k xN k The convolution kernel will be expanded to (N k ,N k ,N) size, if the output is , then the total computational cost t is:

[0100]

[0101] It should be noted that traditional convolution uses an end-to-end processing model for images with multi-channel input and output. This approach has the advantages of reducing the number of convolutions and network complexity, and can alleviate the gradient descent problem associated with increasing network depth. However, this approach increases the network's computational load. Specifically, the cost of a single convolution is affected by the number of channels in both the input and output images, and increases exponentially.

[0102] To reduce algorithm computational overhead and ensure detection algorithm accuracy, this patent innovatively proposes a fast deformable convolution method for solder defect detection to meet actual industrial needs. To address this issue, the present invention innovatively decomposes the convolution process, changing the end-to-end strategy to divide the convolution process into: an offset convolution process that calculates the subsequent convolution kernel deformation offset; a single-channel deformable convolution process that rapidly captures the chromaticity and morphological information of any channel; and a cross-channel joint convolution process that performs collaborative information exchange on each channel.

[0103] First, we construct the convolution kernel K for the offset convolution process operator δ(·) δ , the size is ( ,N,2), thus defining the offset convolution process δ(p0) as follows:

[0104]

[0105] In the above formula and are the weights of the convolution kernel W channel and H channel respectively, p0 is the convolution kernel input pixel point, that is, any pixel position in the input image I, p δ is the convolution kernel K δ The position enumeration of the elements in . By convolving the input image with the above formula, we can obtain the W-direction offset and H-direction offset of each pixel in the image. This offset will be used in the subsequent single-channel deformable convolution process, making the convolution kernel shape adaptively change with the input image to better capture irregular defects on the eutectic solder surface.

[0106] This is followed by the single-channel deformable convolution process operator ξ n (·) Construct several convolution kernels K ξ , the size is , the number of convolution kernels is the number of input image channels N, and the offset obtained by the offset convolution process is applied to the convolution kernel to produce convolution kernel deformation, thereby defining the single-channel deformation convolution process ξ n (p0) is as follows:

[0107]

[0108] In the above formula, n is the channel set of the input image R [R1,...R N ] Any enumeration of the number of channels. is the convolution kernel K of channel n ξ In p ξ The weight at (X(p0,p ξ ),Y(p0,p ξ )) is based on (p0,p ξ ) The calculated convolution kernel K ξ In p ξ The deformation offset at point p(x) and p(y) are the x-axis coordinate and y-axis coordinate of point p respectively. ξ ))p ξ Get the convolution kernel in The deformation is offset, but due to δ(p0+p ξ) The convolution process is not always an integer, so bilinear interpolation G(·) is used for numerical fitting and scaling. That is, bilinear interpolation is performed on the four nearest neighbors of the elements in G(·) to obtain the expected value of the non-integer point.

[0109] The above method can realize separate deformable convolution for each channel, reducing the convolution operation overhead by reducing the size of the convolution kernel, but the data between the channels do not interact collaboratively, which will cause a decrease in network accuracy.

[0110] Therefore, to solve this problem, a channel joint convolution kernel of size (1, 1, N) is constructed for the cross-channel joint convolution operator Y(·), and the fast deformable convolution algorithm is defined as shown in the following formula:

[0111]

[0112] In the above formula, ω n is the channel joint convolution kernel weight of channel n. Through this convolution, multiple channels can be jointly convolved into a single channel to achieve information interaction. Therefore, if the network output is channels, you should use The convolution kernel size is (1,1,N).

[0113] To sum up, the embodiment of the present invention starts from the step-by-step convolution and originally proposes a fast deformable convolution method, which reduces the computational overhead of the convolution process with single-channel convolution and cross-channel joint convolution methods as the core, and relies on the offset convolution method to calculate the offset, which acts on the single-channel convolution process to realize single-channel deformable convolution, so as to improve information extraction and compression effects, thereby improving network accuracy and convergence speed.

[0114] It can be proven that the method of the embodiment of the present invention significantly reduces the convolution operation time:

[0115] For an image R(W,H,N) with a width of W, a height of H, and a number of channels of N, if the fast deformable convolution method is used, if the output is , then the total computational cost t is:

[0116]

[0117] Let's take N k =3, The time overhead of traditional convolution is 81WH, and the time overhead of fast deformation convolution is 42WH.

[0118] The fast deformable convolution method captures the shape representation of solder defects by constructing a deformable convolution process, and decomposes the subsequent convolution process into single-channel convolution and multi-channel joint convolution, thereby improving computational efficiency. This reduces the convolution time while implementing the deformable convolution kernel, and allows for the rapid capture of irregular solder surface defects. The theoretical accuracy and speed are both higher than those of traditional convolution methods, ensuring the practical value of this method.

[0119] Based on the above convolution method, this paper improves the YOLOv5 network structure for solder defect recognition and detection and proposes a fast deformation convolutional network. The network structure is as follows:

[0120] (1) Input: This includes the calculation of adaptive anchor frames for the input image and adaptive image scaling. To improve the network’s ability to recognize small objects, three small-area anchor frames are added to the original nine anchor frames in YOLOv5 to achieve accurate detection of small object defects.

[0121] (2) Backbone: This layer includes the DFocus, DBL, and DSP structures based on fast deformable convolution. DBL uses a fast deformable convolution layer connected to the BN layer and uses Leaky ReLU as the activation function. The DFocus and DSP structures replace the CBL with the DBL based on the original Focus and CSP1 structures, thereby achieving deformation aggregation at different image granularities to form image features and extract the morphological characteristics of irregular defects.

[0122] (3) Neck: This layer adopts the FPN+PAN structure and introduces four times downsampling based on the original YOLOv5, corresponding to the three newly added anchor frames, so as to achieve the purpose of increasing the receptive field and obtaining the feature information of small targets.

[0123] (4) Prediction: YOLOv5 uses GIoU to calculate the positioning loss. Compared with IoU, which only focuses on the overlapping parts, GIoU also focuses on other non-overlapping areas, which can better reflect the overlap rate between the recognition box and the target. However, this loss cannot reflect the regression of the target box. Therefore, this paper uses CIoU as the positioning loss function to better obtain the regression relationship of the target, thereby accelerating network convergence and obtaining more accurate defect localization.

[0124] This method enables direct acquisition of low-light enhanced images from optical sensors in non-uniform lighting environments through image enhancement. Data augmentation algorithms are then used to generate large pseudo datasets. By pre-training a neural network with these pseudo datasets and subsequently training it with real datasets, a highly accurate eutectic solder defect detection network can be developed.

[0125] Example 3

[0126] like Figure 1 As shown, the data preprocessing part of the embodiment of the present invention is as follows:

[0127] Step S201 : Acquire eutectic solder images and manually annotate image defect categories and location information to obtain a real training set.

[0128] In step S202 , low-light enhancement is performed on the image using a non-balanced multi-scale illumination enhancement method to mitigate the impact of uneven illumination.

[0129] In step S203 , the image data is binarized using the Otsu method to convert the solder area into white. The image is scanned to extract the solder center position (x, y) and the deflection angle α.

[0130] In step S204, an affine transformation is performed on the image based on the above center position and deflection angle. The deflection angle (x, y) is positioned at the center of the image by rotation and translation. The deflection angle α is set to 0, and the nearest neighbor edge is used for filling to cover the original real data set.

[0131] Step S205 , classifying the images: images with solder defects are considered defective, otherwise they are considered normal.

[0132] In step S206, K-means clustering is performed on the defective images, with a k value of 5, so that there are a total of six types of images after adding the normal images.

[0133] Step S207 , randomly crop and scale all real data sets to obtain four regions: upper left, lower left, upper right, and lower right.

[0134] Step S208: Randomly stitch the images in each region. The stitching result must meet the following conditions: the generated result includes and only includes the top left, bottom left, top right, and bottom right regions from different images; and the generated regions must be from the same category. The generated images are added to the dummy dataset.

[0135] Step S209: Randomly select a normal image and a defective image, and make them transparent. The transparency of the two images must satisfy the affine transformation premise that the sum is 1. The image label is the same as the image with the lowest transparency, and the generated image is added to the pseudo dataset.

[0136] Step S210 , randomly cropping a defective area from the defective image and overlaying it on the same position of a random normal image to generate an image to be added to a pseudo data set.

[0137] like Figure 2 As shown, the fast deformation convolution part of the embodiment of the present invention is as follows:

[0138] For the input three-dimensional RGB image R, it is first input to the offset convolution part to obtain a two-dimensional offset map, the first dimension of which is the X-direction offset and the second dimension is the Y-direction offset.

[0139] The three-dimensional RGB image R and the two-dimensional offset map are jointly input into the single-channel deformable convolution part, and deformable convolution is performed on the RGB channels respectively, so that the network can heterogeneously capture the structural abnormal parts under different color channels to realize the perception of solder defects on the chip surface.

[0140] The single-channel deformable convolution results are input in parallel to the joint convolution part, and the convolution kernel of size (1,1,3) interacts with the information between channels to achieve the fusion of multi-channel structure and chromaticity information, improve the network accuracy, and then obtain the output.

[0141] like Figure 3 As shown, the network input of the network structure of the embodiment of the present invention adopts a three-segment coding architecture, and encodes the input image through alternating links of the DBL layer and the DSP layer to obtain coding results within different receptive fields, and compress the input information to improve the generalization of the network. The network hidden module adopts a progressive upsampling architecture to restore the low-resolution coded information to realize the joint process of encoding and decoding, and realizes data interaction between receptive fields through cross-linking to improve network accuracy. The network adopts four-module output to achieve accurate recognition under different areas, and has adaptability to situations such as large-area solder unevenness and small-area solder contamination.

[0142] like Figure 4 As shown, the network training process of the embodiment of the present invention is as follows:

[0143] In step S401, the network parameters are initialized based on the network structure. The network weights are randomly assigned, the judgment threshold is 0.8, the learning rate is set to linear, the initial value is 0.3, the final weight is 0.02, the maximum epoch is 700k, the batch size is set to 64 or larger based on the GPU memory, EMA is introduced to adjust the network weights, and the dropout rate is 0.3.

[0144] Step S402: Based on the data preprocessing process, a training set is obtained and divided into a training set and a validation set. The validation set size should be no less than 100 images to ensure the validation effect.

[0145] In step S403, the pseudo dataset is input into the neural network, the CIoU loss is calculated based on the output results and the true labels, the loss is back-propagated based on the gradient descent method, the weight descent gradient is calculated, and the network weights are modified until the network converges on the validation set performance or completes all epochs.

[0146] Step S404: Train the network based on the real training set to improve network performance until the network converges on the validation set or completes all epochs.

[0147] like Figure 5 As shown, the overall process of the method in the embodiment of the present invention is as follows:

[0148] Step S501 : Under constant lighting conditions, an image of a substrate to be inspected is captured using a monocular camera.

[0149] Step S502 , performing Gaussian filtering enhancement on the input substrate image to remove noise caused by instrument influence, environmental factors, etc. The Gaussian filtering preset standard deviation is between 1.5 and 2.0, and the standard deviation value is adjusted according to the camera intrinsic parameter interference.

[0150] Step S503 : low-light enhancement is performed on the image using a non-balanced multi-scale illumination enhancement method to enhance image edge details and mitigate the impact of uneven illumination.

[0151] In step S504, the background and the solder are separated by binarization using the Otsu method, the position of the solder body is obtained through opening operation and connected domain calculation, and the solder posture is adjusted to the standard posture by relying on the nearest neighbor interpolation affine transformation.

[0152] Step S505: input the processed image into the neural network to obtain the result.

[0153] Step S506 , jointly determining whether the eutectic solder has defects based on the classification confidence, classification category, detection frame size, and detection frame position.

[0154] Step S507: output the defective solder position and defect probability.

[0155] Reference Figure 6 The solder defect detection system based on small target recognition proposed in an embodiment of the present invention includes a memory 10, a processor 20, and a computer program stored on the memory 10 and executable on the processor 20, wherein the processor 20 implements the steps of the solder defect detection method based on small target recognition proposed in this embodiment when executing the computer program.

[0156] The specific working process and working principle of the solder defect detection system based on small target recognition of this embodiment can refer to the working process and working principle of the solder defect detection method based on small target recognition of this embodiment.

[0157] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A solder defect detection method based on small target recognition, characterized in that: The method comprises: Collecting solder images and performing image enhancement on the solder images; Obtaining a pseudo data set of solder images, wherein obtaining the pseudo data set of solder images includes: Performing defect marking on the solder images in the solder image set, wherein the defect marking specifically includes marking the category, coordinates, and length and width values ​​of the defects; The solder position of the solder image is extracted using Otsu's binarization method, and the chip is rectified by nearest neighbor filling affine transformation to move the solder to the center of the solder image. Solder images are divided into normal images and defective images according to whether they have defects, and the K-means clustering method is used on the defective images to obtain classified solder images; Randomly crop the classified solder image to obtain a random cropping area; After randomly cropping the solder image set, the randomly cropped regions belonging to the same class are randomly spliced ​​and scaled to obtain a pseudo dataset of solder images; A fast deformable convolution kernel is constructed, and based on the fast deformable convolution kernel, a YOLOv5 neural network is used to construct a solder defect detection network. The specific formula for constructing the fast deformable convolution kernel is: Among them, Y(p0) represents the result of convolution operation on pixel p0 in the solder image, ω n represents the weight of channel n, N represents the total number of solder image channels, and N = 3, ξ n () is a single-channel deformable convolution operator for channel n, p ξ The convolution kernel K of the single-channel deformable convolution ξ The position enumerator of the elements, K ξ The set of weights at channel n, R n is the channel n component of the solder image, G is the bilinear interpolation function, X(p0,p ξ ),Y(p0,p ξ ) are based on (p0,p ξ ) The calculated convolution kernel offset in the X and Y directions, p δ is the convolution kernel K of the offset convolution δ The position enumerator of the elements, and K δ In the weight set of channel w and channel h, p0(x) and p0(y) represent the coordinates of p0 in the X and Y directions respectively. ξ (x) and p ξ (y) represents p ξ Coordinates in the X and Y directions; A pseudo dataset of solder images is used to train a solder defect detection network, and the trained solder defect detection network is used to identify solder defects.

2. The solder defect detection method based on small target recognition according to claim 1, characterized in that: Image enhancement of solder images includes: determining candidate edge regions of the solder image; Calculating an edge coefficient of the candidate edge region, where the edge coefficient is used to represent a degree of fluctuation of pixels within the candidate edge region; According to the edge coefficient, the scale weight of each channel of the solder image is calculated; According to the scale weight of each channel of the solder image, a multi-scale illumination enhancement algorithm is used to enhance the solder image.

3. The solder defect detection method based on small target recognition according to claim 2, characterized in that: The calculation formula of the edge coefficient is: Among them, β n (x, y) represents the edge coefficient of the candidate edge region with the center coordinate (x, y) in channel n, H n (x,y) represents the candidate edge region of channel n, S(H n (x,y)) represents the candidate edge region H in channel n n The standard deviation of all pixels in (x,y), σ n Indicates the standard deviation of channel n, Max(σ n ) and Min(σ n ) represent σ n The maximum and minimum values ​​of .

4. The solder defect detection method based on small target recognition according to claim 3 is characterized in that: According to the edge coefficient, the calculation formula for the scale weight of each channel of the solder image is: Among them, ω n (x,y) represents the scale weight of the pixel with coordinates (x,y) in channel n, β n (x,y) represents the edge coefficient of the candidate edge region with the center coordinate (x,y) in channel n, σ n represents the standard deviation of channel n, σ i represents the standard deviation of channel i, N represents the total number of channels of the solder image, and N=3, i and n are both positive integers greater than 0 and less than 4.

5. The solder defect detection method based on small target recognition according to claim 4, characterized in that: The pseudo dataset for obtaining solder images includes: Randomly extracting a normal image and a defective image, performing transparency merging, and obtaining a first merged image, wherein the transparency of the normal image and the defective image is a random value and the sum is 1; calculating the transparency of the first merged image, and classifying the first merged image into the same category as the image with low transparency among the normal image and the defective image; According to the first merged image determined by the category, a pseudo dataset of solder images is obtained.

6. A solder defect detection system based on small target recognition, the system comprising: A memory (10), a processor (20), and a computer program stored in the memory (10) and executable on the processor (20), wherein the processor (20) implements the steps of the method according to any one of claims 1 to 5 when executing the computer program.

Images