Printed circuit board welding spot defect detection method and system

Abstract

The invention discloses a printed circuit board welding spot defect detection method and system, and the method comprises the steps: obtaining a heat distribution diagram of a printed circuit board in the periodic read-write process of a printed circuit board chip, inputting the heat distribution diagram into a defect detection model, and obtaining a position mark of a welding spot defect of the printed circuit board; the defect detection model is a Faster R-CNN model, Resnet and an FSSD network are used for feature extraction, and the FSSD network is connected between the two Resnet; and a CA attention module is connected behind the ROI pooling pooling layer. Before the image is input into the defect detection model, the definition of the image is improved through operations such as noise removal, the information richness is enhanced through image fusion processing, and the detection effect is effectively improved in combination with the improved defect detection model.

Description

Technical Field

[0001] The invention relates to the technical field of automatic detection of printed circuit boards, in particular to a method and system for detecting defects in solder joints of printed circuit boards. Background Art

[0002] With the rapid development of electronic technology, electronic products are constantly developing in the direction of miniaturization, high performance and multi-function. As the core component of electronic products, the quality and reliability of PCBA (Printed Circuit Board Assembly) directly affect the performance and life of the entire electronic product. In the production process of PCBA, the quality of solder joints is crucial, and any solder joint defects may cause electronic products to fail. On the one hand, there are many solder joints on PCBA and they are densely distributed, making it difficult to clearly distinguish solder joint defects from the surrounding environment. At the same time, there are various types of solder joint defects, including cold solder joints, leaking solder joints, cold solder joints, too much or too little solder, missing holes, rat bites, open circuits, short circuits, bone spurs and fake copper. On the other hand, for relatively small targets such as solder joints, the detection accuracy requirements are extremely high.

[0003] The appearance of PCBA solder joints is diverse, and the production environment is also special. The traditional PCBA solder joint defect detection system and method have weak noise resistance and insufficient generalization, and have gradually failed to meet actual needs. Therefore, there is an urgent need for a PCBA solder joint defect detection solution that can solve the low recognition accuracy caused by factors such as the diversity of PCBA solder joint appearance, uneven quality of collected images, and large noise interference. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a method and system for detecting solder joint defects on a printed circuit board with high detection accuracy.

[0005] Technical solution: The printed circuit board solder joint defect detection method described in the present invention includes the following steps: allowing the printed circuit board chip to be read and written periodically, obtaining a heat distribution map of the printed circuit board during the reading and writing process, inputting the heat distribution map into a defect detection model, and obtaining a position mark of the solder joint defect of the printed circuit board; the defect detection model is a Faster R-CNN model, in which Resnet and FSSD networks are used for feature extraction, and an FSSD network is connected between two Resnets; and a CA attention module is connected after the ROI pooling layer.

[0006] Furthermore, obtaining the heat distribution map of the printed circuit board during the reading and writing process includes: during the reading and writing process of the chip on the printed circuit board, using a terminal device to collect a number of heat distribution maps from multiple angles for the solder joint area, and uploading the heat distribution map to the edge device after the terminal device performs preliminary processing on the heat distribution map, and the preliminary processing includes image compression processing, noise reduction processing and / or color adjustment.

[0007] Furthermore, in the edge device, the Laplace pyramid algorithm is used to perform image fusion processing on the multiple heat distribution maps at multiple angles, and the fused heat distribution map is input into the defect detection model to obtain the position mark of the solder joint defect of the printed circuit board.

[0008] Furthermore, the cyclical reading and writing of the chip on the printed circuit board includes: applying thermal stimulation to the chip on the printed circuit board to start the cyclical reading and writing, and each bit of the data is flipped in sequence during the reading and writing process.

[0009] Furthermore, the loss function of the defect detection model is L(p,u,t u ,v)=L cls (p,u)+λ[u≥1]L loc (t u ,v);

[0010] Among them, L cls (p,u)=-logp u is the cross entropy loss, L loc is the loss between the classification target value and the quadruple of the detection box, calculated using smoothL1 loss; p is the discrete probability distribution of each candidate region output, u is the classification target value, v is the detection box regression target value, t u is the detection box regression offset and λ is the parameter.

[0011] Furthermore, on a printed circuit board production line, the corresponding printed circuit board is laser marked according to the position mark of the solder joint defect of the printed circuit board, so as to obtain a printed circuit board with a solder joint defect and trigger an alarm.

[0012] The printed circuit board solder joint defect detection system of the present invention comprises:

[0013] An image acquisition unit is used to read and write the chip on the printed circuit board periodically, and obtain a heat distribution map of the printed circuit board during the reading and writing process;

[0014] The defect detection unit is used to input the heat distribution map into the defect detection model to obtain the position mark of the solder joint defect of the printed circuit board; the defect detection model is a Faster R-CNN model, in which Resnet and FSSD networks are used for feature extraction, and a FSSD network is connected between two Resnets; and a CA attention module is connected after the ROI pooling layer.

[0015] The printed circuit board solder joint defect detection device of the present invention comprises an image acquisition device, a motion control system and a background operating system;

[0016] Image acquisition equipment includes industrial cameras, infrared thermal imagers, light sources and image acquisition cards to collect thermal distribution maps of printed circuit boards

[0017] The motion control system includes a first motor for controlling the extension and retraction of a detection table, on which a printed circuit board to be detected is placed; a second motor for controlling the position of an image acquisition device; and a sensor for adjusting the speed of the first motor and the second motor according to the detection speed;

[0018] The background operating system is used to upload the collected printed circuit board heat distribution map to the edge device, perform detection according to the printed circuit board solder joint defect detection method, and upload the defective printed circuit board heat distribution map and defect detection results to the cloud server.

[0019] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the printed circuit board solder joint defect detection method is implemented.

[0020] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the printed circuit board solder joint defect detection method is implemented.

[0021] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) The Faster R-CNN model has the characteristics of high detection accuracy, fast speed and good detection effect on small targets. Through the special RPN network in its network structure, the influence of image background interference can be better improved, and the convolution operation is optimized at the same time, the detection efficiency and accuracy are improved, so as to better realize the detection of PCBA solder joint defects; by optimizing the feature extraction network part of Faster R-CNN, a light and lightweight feature fusion module is adopted to improve the detection speed and accuracy; by adding an attention mechanism after the ROI pooling layer, the important feature channels in different ROIs are emphasized, so that the network pays more attention to the target to be detected, so as to improve the detection effect; (2) The present invention improves the clarity of the image by removing noise and other operations. For example, multi-focus image fusion can obtain a fully focused clear image. Introducing image fusion for image preprocessing before image detection can enhance information richness and fuse images with more comprehensive information. At the same time, image fusion enhances the reliability and stability of the image, improves robustness, and reduces the uncertainty of a single image caused by various factors. Even if a problem occurs in a certain image source, the fused image can still rely on the information of other image sources to maintain a certain reliability; (3) The present invention uses a cloud-edge system, which can achieve low latency and reduce cloud pressure in terms of data processing and response; in terms of reliability and availability, it has offline operation capabilities and adopts a distributed architecture to avoid single point failures; it can optimize resource utilization, flexibly allocate resources and save energy and reduce emissions; it also improves security and privacy protection, and reduces risks through local data processing and fine access control. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a structural diagram of the improved Faster R-CNN defect detection model of the present invention;

[0023] Figure 2 It is a complete structural diagram of ResNet and FPN of the present invention;

[0024] Figure 3 is a structural diagram of the FSSD of the present invention;

[0025] Figure 4 A schematic diagram of a fixed reference frame with different scales and aspect ratios of anchors in an embodiment of the present invention;

[0026] Figure 5 The RPN split flow chart of the present invention;

[0027] Figure 6 It is the flow chart of the ROI pooling layer of the present invention;

[0028] Figure 7 This is the structural diagram of the CA attention mechanism of the present invention;

[0029] Figure 8 It is the Classification and Regression classification regression network structure diagram of the present invention;

[0030] Fig. 9 Schematic diagram of the cloud-edge-end system of the present invention;

[0031] Fig.10 Schematic diagram of the Laplace pyramid of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0033] The printed circuit board solder joint defect detection method comprises the following steps:

[0034] Step 1: Create a solder joint defect detection dataset.

[0035] Design a program to store and read data from PCB chips. To make the PCB temperature data change significantly, the data written to the storage PCB is set to a larger value, and each bit of the data is flipped in turn during the cyclic reading and writing process. Use an infrared thermal imager to capture the changes in thermal characteristics during the PCB reading and writing process, analyze the working status of the solder joints of the PCB chips, and extract multi-region thermal signals from the infrared image to obtain the working heat distribution map of the PCB chips.

[0036] The reading and writing process consists of two parts:

[0037] (1) Design a program to store and read data from the chip on the PCB. The chip write timing is when the FPGA controller's "data reception ready status flag" and "command reception ready status flag" are both pulled high to prepare to receive the current data; the clock pulls up the "write enable flag" to execute the write operation. The chip read timing is when the chip controller's "command reception ready flag" is pulled high to start reading chip data.

[0038] (2) FPGA board control reading and writing. The FPGA board is powered on to apply thermal stimulation and solidify the PCB reading and writing program. Then the chip starts cyclic reading and writing and waits for the reading and writing to end. In order to make the PCB temperature data change significantly, the data written to the storage PCB is set to a larger value, and each bit of the data is flipped in turn during the cyclic reading and writing process.

[0039] The process of extracting the heat distribution map includes: placing the chip area of ​​the FPGA board to be tested in the field of view of the infrared thermal imager, and the infrared thermal imager captures the changes in thermal characteristics during the reading and writing process of the PCB. Since it is impossible to visually determine whether the PCB has defects during the reading and writing process, multi-region thermal signal extraction is performed on the infrared image. Compared with the normal PCB thermal signal, the temperature changes during the periodic reading and writing process of the faulty FPGA board are significantly different, and it can be preliminarily distinguished.

[0040] The dataset establishment process includes: obtaining multiple sets of heat distribution map images, dividing the heat map dataset into a heat map training set and a heat map test set according to the ratio of 8:2.

[0041] Step 2: Build a defect detection model.

[0042] An improved Faster R-CNN defect detection model is established. The FSSD network is introduced into Faster R-CNN to extract input image features and achieve local optimization of Faster-RCNN. The CA attention mechanism is introduced into the Faster R-CNN neural network model.

[0043] The improved Faster R-CNN model is divided into four modules, including Conv layers network, Region proposal Network network, ROI pooling layer and Classification and Regression classification regression, such as Figure 1 shown.

[0044] ① Conv layers feature extraction network: The traditional Faster R-CNN uses the ResNet+FPN structure to extract features. The traditional Faster R-CNN only needs to input one feature map into the subsequent network. Due to the addition of the FPN structure, multiple feature maps need to be sent to the subsequent network one by one for processing. Resnet performs feature extraction, and the subsequent FPN structure allows the feature map of the current layer to be integrated with the features of the future layer for upsampling and utilization. Because of such a structure, the current feature map can obtain the information of the future layer, which means that the low-order features and high-order features are organically integrated to improve the detection accuracy.

[0045] The complete structure of ResNet and FPN is as follows Figure 2 As shown in the figure, after Resnet performs feature extraction and the FPN network performs feature fusion to obtain multiple feature maps, the feature map input to the RPN network is [p2, p3, p4, p5, p6], and the input to the subsequent target detection network Faster R-CNN is [p2, p3, p4, p5].

[0046] The present invention introduces the FSSD network in the feature extraction network part to replace the FPN network for feature extraction. Figure 3 This is the structural diagram of FSSD, in which the feature fusion module can project and splice features of different scales, then use the batch normalization layer to normalize the feature values, and then add some downsampling blocks to generate a new feature pyramid, which is fed back to the multi-box detector to generate the final detection result to make full use of the features. Its fusion feature module can be described as:

[0047]

[0048] Explain the meaning of the letters in the formula: is the source feature map to be fused, represents the transformation function before concatenation of each source feature map, φ f represents the feature fusion function, φ p Represents the function for generating feature pyramid, φ c,l Represents a method for predicting targets from a provided feature pyramid. Represents the range of the layer that should be fused, loc (location) represents the positioning information, usually the position and size of the bounding box, and class (classification) represents the classification information, usually the probability distribution of the category to which each detected target belongs.

[0049] ②Region proposal network: Classic detection methods are very time-consuming to generate detection frames, such as using sliding windows to generate detection frames in Overfeat, or using selective search methods to generate detection frames in R-CNN. Faster R-CNN abandons the traditional sliding window and selective search methods and directly uses RPN to generate candidate regions, which can greatly improve the detection speed. This module can determine whether the anchor is foreground or background based on the region of interest, which can effectively distinguish defects on the circuit board from the background and improve the detection accuracy.

[0050] The main process of the RPN network is:

[0051] 1) Generate a series of fixed reference frame anchors, covering any position of the image, and then send them to the subsequent network for classification and regression;

[0052] 2) Classification branch: Use softmax classification to determine whether the anchor contains the target;

[0053] 3) Regression branch: calculate the offset of the target box to the anchors to obtain accurate candidate regions;

[0054] 4) The final Proposal layer is responsible for obtaining candidate regions by integrating the anchors containing the target and the corresponding bbox regression offset, while eliminating candidate regions that are too small or beyond the boundary.

[0055] Furthermore, the anchors represent fixed reference frames in target detection, first as Figure 4 A set of fixed reference frames of different scales and aspect ratios are preset to cover almost all positions. Each reference frame is responsible for detecting targets whose intersection-over-union ratio is greater than a threshold (preset training value, usually 0.5 or 0.7). The anchor technology converts the candidate region generation problem into "is there a target in this fixed reference frame, and how far the target frame deviates from the reference frame?" There is no need to traverse the sliding window at multiple scales, which truly achieves good and fast results.

[0056] In the original Faster R-CNN, due to the FPN network, anchors are generated in multiple feature maps of different scales. Assuming that the size of a feature map is h*w, the stride of the downsampling multiple of this feature relative to the input image is first calculated:

[0057]

[0058] Anchors of different ratios are generated on each scale feature map: After obtaining a series of anchors, they can be sent to the subsequent network for classification and regression.

[0059] The FSSD network introduced in the present invention can fuse features of different scales, generate a new feature pyramid through some downsampling blocks, and then perform anchor operations on different scales, which can improve the detection speed while ensuring accuracy.

[0060] The RPN classification process is as follows Figure 5 As shown in the figure: After a matrix of size M*N is sent to the Faster R-CNN network, it is extracted through Conv layers and converted into a feature map of size H*W in the RPN network. After that, a 1*1 convolution is performed to obtain a feature map of size [batchsize, H, W, 18]. Then, a deformation is performed to convert the feature map into a feature map of size [batchsize, 9*H, W, 2]. After that, it is sent to the softmax for classification. After obtaining the classification result, the reshape is performed again to finally obtain a result of size [batchsize, H, W, 18]. 18 represents the probability value of whether k=9 anchors contain the target.

[0061] The RPN regression part outputs a feature map of [1, H, W, 4*9] after convolution, which is equivalent to each point of featuremaps having 9 anchors, and each anchor has 4 transformations for regression:

[0062] [d x (A),d y (A),d w (A),d h (A)] (5)

[0063] This transformation predicts the translation and scale factor between the anchor and the true value:

[0064]

[0065] t h =log(h / h a ) (9)

[0066] The Proposal layer is responsible for integrating the results of the RPN network on the classification and regression of the anchors, using the regression results to modify the anchors containing the target, calculate the candidate region, and send it to the subsequent ROIPooling layer. The processing flow of the Proposal layer is as follows:

[0067] 1) Use the results of RPN network regression to correct all anchors and obtain the corrected detection frame;

[0068] 2) Sort the detection boxes from large to small according to the probability values ​​of the softmax output of the RPN network classification, and extract the first 6000 results, that is, extract the detection boxes after the corrected position;

[0069] 3) The detection box that exceeds the image boundary is limited to the image boundary to prevent the candidate area from exceeding the image boundary during subsequent ROI pooling;

[0070] 4) Perform non-maximum suppression (NMS) on the remaining detection boxes;

[0071] 5) The output of the proposal layer is the normalized coordinate value [x1, y1, x2, y2] corresponding to the scale of the input network image.

[0072] ③ROI pooling layer: The ROI pooling layer is responsible for collecting candidate regions generated by the RPN network, mapping them to feature maps and fixing the dimensions, and sending them to the subsequent network for classification and regression. The process is as follows Figure 6As shown in the figure, the ROI pooling layer uses max pooling to convert the features within any valid ROI region into a small feature map with a fixed spatial range of pool_H×pool_W, where pool_H and pool_W are hyperparameters, such as set to 7*7, and they are independent of any specific ROI. During implementation, the previous FSSD network generates multiple scale feature maps. Here, ROIs of different scales use different feature layers as the input of the ROI pooling layer. For large-scale ROIs, use some later pyramid layers, such as P5; for small-scale ROIs, use some earlier feature layers, such as P3. The following formula is used to determine the feature layer where the ROI is located:

[0073]

[0074] where 224 is the standard input of ImageNet, k 0 is the base value, set to 4, w and h are the length and width of the ROI region. Assuming the ROI is 112*112 in size, then k = k 0 -1 = 4 - 1 = 3, which means that this ROI should use the feature layer of P3. The k value will be rounded to prevent the result from being a non-integer, and in order to ensure that the k value is between 2 and 5, truncation processing will also be performed. The output result is that each candidate region is fixed to a size of 7*7.

[0075] The implementation of the CA attention mechanism, as Figure 7 shown, can be considered as two parallel stages. Specifically: the input feature map Input with size C*H*W is pooled in the X and Y directions respectively, resulting in feature maps with sizes C*H*1 and C*1*W respectively.

[0076] The output at height h of the c-th channel can be expressed as:

[0077]

[0078] The output at width w of the c-th channel can be expressed as:

[0079]

[0080] The above two transformations aggregate features along two spatial directions respectively, obtaining a pair of direction-aware feature maps. This is very different from the compression operation that generates a single feature vector in the channel attention method. These two transformations also enable the attention module to capture long-range dependencies along one spatial direction and retain precise position information along the other spatial direction, thereby helping the network to more accurately locate the object of interest.

[0081] Next, the feature maps obtained by formula (11) and formula (12) are concatenated and sent to a shared 1*1 convolution transformation function F1, resulting in:

[0082] f=δ(F1(z h ,z w )) (13)

[0083] Among them, [z h ,z w ] represents the connection operation along the spatial dimension, δ is the nonlinear activation function, and formula (14) is the intermediate feature map, which encodes spatial information in the horizontal and vertical directions. r is the reduction ratio of the control block size.

[0084] f∈R C / r×(H+W) (14)

[0085] Then, we split f into two separate tensors along the spatial dimension:

[0086] f h ∈R C / r×H (15)

[0087] f w ∈R C / r×W (16)

[0088] Use two 1*1 convolution operations to increase the dimension of f h and f w Convert it to a tensor with the same number of channels as the input, and then combine it with the sigmoid activation function (σ) to get:

[0089] g h =σ(F h (f h )) (17)

[0090] g w =σ(F w (f w )) (18)

[0091] Finally: The output formula of CoordinateAttention can be written as:

[0092]

[0093] The present invention adds an attention mechanism after the ROI pooling layer of the improved Faster R-CNN model to emphasize the important feature channels in different ROIs, so that the network pays more attention to the target to be detected, thereby achieving the purpose of improving the detection effect.

[0094] ④Classification and Regression: This part uses the feature map of the candidate area to calculate the defect category of each candidate area through the fully connected layer and softmax, and outputs the probability value; at the same time, the regression method is used again to obtain the position offset of each candidate area for regressing a more accurate target detection frame. The network structure of this part is as follows Figure 8 After the ROI pooling layer obtains the fixed-size feature map, it is sent to the subsequent network to classify the candidate area through full connection and softmax and regress the candidate area again to obtain a higher-precision detection frame.

[0095] Step 3: Training the defect detection model.

[0096] The training of Faster R-CNN is divided into two parts, namely the training of the RPN network and the detection network Faster R-CNN. The entire training process is divided into four steps:

[0097] Step 1: Training of the RPN network, initialized using the ImageNet pre-trained model, and end-to-end fine-tuning for the region proposal task.

[0098] Step 2: Using the proposal boxes generated by the RPN in the first step, Faster R-CNN trains a separate detection network. This detection network is also initialized by the ImageNet pre-trained model. At this time, the two networks do not share convolutional layers.

[0099] Step 3: Initialize RPN training with the detection network, but fix the shared convolutional layers and only fine-tune the RPN-specific layers. Now the two networks share the convolutional layers.

[0100] Step 4: Keep the shared convolutional layers fixed and fine-tune the fc layer of Faster R-CNN. In this way, the two networks share the same convolutional layers and form a unified network.

[0101] Furthermore, the training of the RPN network is: initialized using the ImageNet pre-trained model and end-to-end fine-tuned for the region proposal task. The role of the RPN network is to extract candidate regions that contain the target and are regressed from a large number of anchors. In order to train the RPN, each anchor is assigned a label of whether it contains the target, that is, a label of positive and negative samples, and then trained.

[0102] The positive and negative sample labeling process is as follows: the anchor whose IOU with the real box ground truth (GT) is greater than 0.7 is a positive sample, that is, the anchor contains the target, and the target value is set to 1; the anchor whose IOU with the real box ground truth (GT) is less than 0.3 is a negative sample, that is, the anchor does not contain the target, and the target value is set to -1;

[0103] Other anchors are discarded and do not participate in network training, and their target values ​​are set to 0.

[0104] The loss function of the RPN network is:

[0105]

[0106] Where i represents the index of the anchor, p i is the probability that the i-th anchor is predicted as the target, is the ground-truth label. If the anchor is positive, the ground-truth label is 1, otherwise it is 0. That is, when the IoU between the ith anchor and the GT is greater than 0.7, the anchor is considered to be positive and the label is 1; otherwise, when IoU is less than 0.3, the anchor is considered to be negative and the label is 0. i Represents the four parameterized prediction results of the positive sample anchor to the bounding box of the prediction area. is the offset of the ground-truth box corresponding to this positive anchor, as shown below:

[0107] Predicted value:

[0108]

[0109] True value:

[0110]

[0111] Where x, y, w, and h represent the center coordinates of the window and the width and height of the window. a and x * They represent the coordinates of the prediction window, anchor window, and Ground Truth respectively (same for y, w, and h).

[0112] The entire loss is divided into two parts: classification and regression losses

[0113] The classification loss is the softmax loss of a binary classifier:

[0114]

[0115] is the regression loss, which is the smooth(x) loss, and only positive samples participate in the regression loss calculation:

[0116]

[0117] and are used to normalize the classification loss and regression loss respectively. The default setting is batch size, which is initialized with the number of anchor positions ~2000. If and differ too much, the parameter λ is used to balance the two. The value is generally N. cls and N reg A ratio of g of 10 is sufficient.

[0118] During training, the positive and negative samples of each iteration are composed of the positive and negative samples of an image: 256 anchors are randomly sampled and the loss function is calculated, where the ratio of sampled positive and negative anchors is 1:1; all new layers (layers after the last convolutional layer) are randomly initialized by weights obtained from a Gaussian distribution with zero mean and standard deviation of 0.01, and all other layers (i.e., shared convolutional layers) are initialized by models pre-trained for ImageNet classification; the network is trained using the stochastic gradient descent algorithm with momentum.

[0119] Training of Faster R-CNN network: Use the candidate regions collected by the RPN network and the features extracted by the imageNet pre-trained convolutional network to train the detection Faster R-CNN network.

[0120] The positive and negative sample labeling process is as follows: the candidate area whose IOU with the true box ground truth (GT) is greater than 0.5 is set as a positive sample, and the target value of the category is the category of GT; the candidate area whose IOU with the true box ground truth (GT) is less than 0.5 is set as a negative sample, and the target value of the category is 0.

[0121] The loss function of Faster R-CNN is:

[0122] The output of Faster R-CNN consists of two parts: one is the softmax layer for classification, with k output categories plus the "background" category, and the other is the boundingbox regressor. That is:

[0123] A discrete probability distribution (for each candidate region) over k+1 categories is output as part of the network. Usually, the probability value is calculated by Softmax on the k+1 outputs of the fully connected layer.

[0124] The other part outputs the offset regressed by each detection box in the k categories, where the scale-invariant transformation and logarithmic space height / width shift relative to the candidate box are specified, which is the same as in the RPN network.

[0125] Each candidate region for training is labeled with a classification target value u and a detection box regression target value v. Background samples are represented by u = 0, and a multi-task loss L is used to jointly train classification and detection box regression for each labeled candidate region:

[0126] L(p,u,t u ,v)=L cls (p,u)+λ[u≥1]L loc (t u ,v) (25)

[0127] Among them, represents the cross entropy loss, and the second loss is the loss between the quadruple that defines the target value and the predicted detection box. It is calculated using the smoothL1 loss. Similarly, only the candidate regions of positive samples (non-background) are used to calculate the regression loss. The parameter λ is set to 1, which is the regression offset of the Faster R-CNN output for each detection box in the u categories.

[0128] Faster R-CNN training obtains positive and negative samples in each image: all positive samples are sorted according to the IOU value, the first 256 regions of each image are taken, and the coordinates of these regions are saved as training samples for the image; the weights of the fully connected layers used for Softmax classification and detection box regression are initialized using zero-mean Gaussian distributions with variances 0.01 and 0.001, respectively, and the bias is initialized to 0. The feature extraction network uses the ImageNet pre-trained network; the gradient descent algorithm is used for optimization.

[0129] Step 4: Obtain an infrared image of the PCB to be inspected, perform preprocessing, and perform defect detection using a defect detection model in conjunction with the cloud-edge system. The cloud-edge system is as follows: Fig. 9 shown.

[0130] (1) Image acquisition and preliminary processing of terminal equipment: On the PCBA board production line, high-precision industrial cameras are used as terminal equipment to acquire images from multiple angles in the solder joint area. The acquired heat distribution map will be processed initially on the terminal equipment. First, image compression is performed to reduce the amount of data transmission. At the same time, simple noise reduction processing is performed to remove some slight noise caused by factors such as the production environment. In addition, preliminary color adjustments are performed to enhance the contrast between the solder joint and the surrounding area, making the solder joint more prominent.

[0131] (2) Real-time analysis and decision-making by edge devices: The terminal device transmits the heat distribution map to the edge device. To ensure the efficiency and stability of data transmission, low-latency wireless transmission technology is used. After receiving the image, the edge device performs image fusion processing, and then uses the trained defect detection model to perform real-time analysis to determine whether there are defects in the solder joints, such as cold solder joints, leaking solder joints, too much or too little solder, etc. Based on the analysis results, the edge device can make decisions immediately. If a solder joint defect is detected, the edge device can trigger an alarm to achieve product sorting. At the same time, the edge device uploads the heat distribution map and analysis results to the cloud for more in-depth processing and long-term storage.

[0132] Image fusion processing is to fuse several heat distribution maps using the Laplacian pyramid algorithm. In the Gaussian pyramid operation process, the image will lose some high-frequency detail information after convolution and downsampling operations. In order to describe this high-frequency information, the Laplacian pyramid is defined.

[0133] First, the original image O is downsampled, then upsampled to get Odu, and finally subtracted from the original image to get the Laplacian pyramid image, as shown in Fig.10 shown.

[0134] Use the original image to insert the formula O-Odu to get the 0th level of the Laplacian pyramid.

[0135] Use the original image to downsample to get Od, downsample it twice to get O2d, and then upsample it to get O2du. Substitute it into the formula Od-O2du to get the first layer of the Laplacian pyramid.

[0136] Finally, the images of different layers are fused to obtain the final fused image.

[0137] (3) In-depth analysis and feedback from cloud servers: After receiving the heat distribution map and analysis results from the edge device, the cloud server performs data verification and storage backup to ensure data security and reliability. In the cloud, a large amount of solder joint image data is deeply analyzed. Using big data analysis technology, the quality trends and potential problems of solder joints of PCBA boards in different time periods and batches are analyzed to provide a strong basis for the continuous optimization of the production process. At the same time, the defect detection model is further updated and optimized, and the algorithm's detection accuracy and efficiency of various solder joint defects are improved by continuously learning new heat distribution maps. The cloud server feeds back the processing results to the edge device and end users, including detailed solder joint quality analysis reports, production process improvement suggestions, and remote control instructions. In addition, the cloud can also be integrated with the company's production management system to realize intelligent management and decision-making of solder joint quality during PCBA board production, thereby improving the efficiency and product quality of the entire production line.

[0138] The printed circuit board solder joint defect detection system of the present invention comprises:

[0139] An image acquisition unit is used to read and write the chip on the printed circuit board periodically, and obtain a heat distribution map of the printed circuit board during the reading and writing process;

[0140] The defect detection unit is used to input the heat distribution map into the defect detection model to obtain the position mark of the solder joint defect of the printed circuit board; the defect detection model is a Faster R-CNN model, in which Resnet and FSSD networks are used for feature extraction, and a FSSD network is connected between two Resnets; and a CA attention module is connected after the ROI pooling layer.

[0141] The printed circuit board solder joint defect detection device of the present invention comprises an image acquisition device, a motion control system and a background operating system;

[0142] Image acquisition equipment includes industrial cameras, infrared thermal imagers, light sources and image acquisition cards to collect thermal distribution maps of printed circuit boards

[0143] The motion control system includes a first motor for controlling the extension and retraction of a detection table, on which a printed circuit board to be detected is placed; a second motor for controlling the position of an image acquisition device; and a sensor for adjusting the speed of the first motor and the second motor according to the detection speed;

[0144] The background operating system is used to upload the collected printed circuit board heat distribution map to the edge device, perform detection according to the printed circuit board solder joint defect detection method, and upload the defective printed circuit board heat distribution map and defect detection results to the cloud server.

[0145] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the printed circuit board solder joint defect detection method is implemented.

[0146] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the printed circuit board solder joint defect detection method is implemented.

[0147] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store program code in the form of instructions or data structures and that can be accessed by a computer.

[0148] The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.

Claims

1. A method for detecting solder joint defects on a printed circuit board, characterized in that: The method comprises the following steps: making the printed circuit board chip read and write periodically, obtaining the heat distribution map of the printed circuit board during the reading and writing process, inputting the heat distribution map into the defect detection model, and obtaining the position mark of the solder joint defect of the printed circuit board; the defect detection model is a FasterR-CNN model, in which Resnet and FSSD networks are used for feature extraction, and an FSSD network is connected between two Resnets; and a CA attention module is connected after the ROIpooling pooling layer.

2. The printed circuit board solder joint defect detection method according to claim 1, characterized in that: Acquiring a heat distribution diagram of a printed circuit board during the reading and writing process includes: During the reading and writing process of the chip on the printed circuit board, a terminal device is used to collect several heat distribution maps from multiple angles of the solder joint area. After the terminal device performs preliminary processing on the heat distribution map, it is uploaded to the edge device. The preliminary processing includes image compression processing, noise reduction processing and / or color adjustment.

3. The printed circuit board solder joint defect detection method according to claim 2, characterized in that: In the edge device, the Laplace pyramid algorithm is used to perform image fusion processing on the multiple heat distribution maps at multiple angles, and the fused heat distribution map is input into the defect detection model to obtain the position mark of the solder joint defect of the printed circuit board.

4. The printed circuit board solder joint defect detection method according to claim 1, characterized in that: Cycling the chip on the printed circuit board includes: Thermal stimulation is applied to the chip on the printed circuit board to start the cyclic reading and writing process. During the reading and writing process, each bit of data is flipped in sequence.

5. The printed circuit board solder joint defect detection method according to claim 1, characterized in that: The loss function of the defect detection model is L(p,u,t u ,v=L cls (p,u+λ[u≥1L loc (t u ,v); Among them, L cls (p,u)=-logp u is the cross entropy loss, L loc is the loss between the classification target value and the quadruple of the detection box, calculated using smoothL1 loss; p is the discrete probability distribution of each candidate region output, u is the classification target value, v is the detection box regression target value, t u is the detection box regression offset and λ is the parameter.

6. The printed circuit board solder joint defect detection method according to claim 1, characterized in that: On a printed circuit board production line, according to the position mark of the solder joint defect of the printed circuit board, the corresponding printed circuit board is laser marked to obtain a printed circuit board with a solder joint defect and trigger an alarm.

7. A printed circuit board solder joint defect detection system, characterized in that: include: An image acquisition unit is used to read and write the chip on the printed circuit board periodically, and obtain a heat distribution map of the printed circuit board during the reading and writing process; The defect detection unit is used to input the heat distribution map into the defect detection model to obtain the position mark of the solder joint defect of the printed circuit board; the defect detection model is a FasterR-CNN model, in which Resnet and FSSD networks are used for feature extraction, and a FSSD network is connected between two Resnets; and a CA attention module is connected after the ROIpooling pooling layer.

8. A printed circuit board solder joint defect detection device, characterized in that: Including image acquisition equipment, motion control system and background operating system; Image acquisition equipment includes industrial cameras, infrared thermal imagers, light sources and image acquisition cards to collect the thermal distribution map of printed circuit boards The motion control system includes a first motor for controlling the extension and retraction of a detection table, on which a printed circuit board to be detected is placed; a second motor for controlling the position of an image acquisition device; and a sensor for adjusting the speed of the first motor and the second motor according to the detection speed; The background operating system is used to upload the collected thermal distribution map of the printed circuit board to the edge device, perform detection according to the printed circuit board solder joint defect detection method according to any one of claims 1-6, and upload the thermal distribution map of the defective printed circuit board and the defect detection results to the cloud server.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the printed circuit board solder joint defect detection method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the printed circuit board solder joint defect detection method according to any one of claims 1 to 6 is implemented.

Images