Method and device for detecting surface element of PCB (Printed Circuit Board)

Abstract

The invention relates to a PCB surface element detection method and device, and the method comprises the steps: detecting the integrity of a PCB image entering a camera collection image in a current video frame, and obtaining an integrity result; continuously collecting multiple frames of PCB images at fixed time intervals; screening and processing the acquired multiple frames of PCB images to obtain a to-be-detected PCB image; constructing a multi-scale twin space deformation sensing network, and realizing space alignment of the PCB image to be detected and the template image to obtain an aligned PCB image; cutting the aligned PCB image according to the position of the element in the template image to obtain a to-be-detected element; constructing a pseudo-twinning mixed extraction network to realize feature extraction of a to-be-detected element so as to obtain features of the to-be-detected element; constructing a multi-level similarity judgment mechanism, and realizing matching between the features of the to-be-detected element and the features of the element in the template image; according to the invention, accurate alignment and matching of the PCB image to be detected and the template image of the current batch are realized, and missing detection and wrong detection are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition, and in particular to a method and device for detecting components on a PCB surface. Background Art

[0002] PCB surface component inspection is a quality control process that ensures that all electronic components on the PCB are correctly installed and well soldered. This inspection process is very important in the manufacture of electronic products. Therefore, the design of the PCB component inspection system is particularly important to ensure the safety and stability of various electronic products.

[0003] Currently, there are two main technical solutions for PCB component detection: one is to detect PCB components based on traditional methods, but traditional methods mainly rely on artificial design features, and the shape layout of different types of circuit boards varies greatly, so it is difficult to achieve good accuracy and versatility. The other solution is to detect PCB components based on deep learning. This solution has strong versatility and higher accuracy, but with the continuous development of integrated circuits towards miniaturization, the size difference of components on PCBs has gradually increased, making target detection miss or misdetect small-sized components due to weakened features when the size difference is significant, such as: chip resistors, chip capacitors, etc. This missed detection or misdetection will misjudge the quality problems of components on PCBs such as soldering leaks and missoldering.

[0004] Chinese patent with publication number CN116168017A discloses a PCB component detection method, system and storage medium based on deep learning, Chinese patent with publication number CN117745684A discloses a PCB defect detection method and system based on improved YOLOv7 algorithm, and Chinese patent CN 114429445 A discloses a PCB defect detection and identification method based on MAIRNet. The above patents all use deep learning target detection methods to detect PCB components, which have good versatility, but they cannot solve the problem of missed detection and wrong detection of small targets in target detection with significant size differences. Summary of the invention

[0005] The present invention provides a method and device for detecting components on the surface of a PCB, which realizes accurate alignment and matching between the image of a PCB to be detected and the image of a template of a current batch, avoids the problems of missed detection and wrong detection of small targets, and completes accurate detection of missed soldering and wrong soldering of electronic components on the PCB.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for detecting components on a PCB surface comprises the following steps:

[0008] Step 1: System initialization;

[0009] Step 2: Capture video frames through the camera;

[0010] Step 3: Detect the completeness of the PCB image in the current video frame entering the camera acquisition screen to obtain the completeness result;

[0011] Step 4: If the completeness result is incomplete, go to step 2, otherwise go to step 5;

[0012] Step 5: Continuously collect multiple frames of PCB images at fixed time intervals;

[0013] Step 6: Screen and process the collected multiple frames of PCB images to obtain the PCB image to be tested;

[0014] Step 7: Construct a multi-scale twin spatial deformation perception network to achieve spatial alignment between the PCB image to be detected and the template image, and obtain the aligned PCB image;

[0015] The input of the multi-scale twin space deformation perception network is the template image and the PCB image to be detected;

[0016] The multi-scale twin spatial deformation perception network consists of two parallel branches. The input of the first branch is the template image. The second branch inputs the template image and the PCB image to be detected into the pre-alignment network to generate a pre-aligned PCB image as the input of the second branch. The two branches successively pass through the multi-scale feature extraction module with shared weights to extract feature maps at scales of 1 / 2, 1 / 4, and 1 / 8 of the input image resolution respectively. The feature maps of the three scales of the two branches are simultaneously input into the spatial deformation perception module to generate fused feature maps of three scales. The fused feature maps of three scales are respectively input into three parallel prediction heads, and the prediction results of the three prediction heads are added to obtain the pixel coordinate offset. The remapping module uses the pixel coordinate offset to transform the PCB image to be detected to obtain the aligned PCB image.

[0017] The pre-aligned network uses an improved STN network, which replaces the ordinary convolution in the STN network with dilated convolution;

[0018] The feature extraction module uses the ENET network, which outputs the encoder stage feature layer as small-scale features, the first upsampling layer as medium-scale features, and the second upsampling layer as large-scale features.

[0019] In the spatial deformation perception module, the 1 / 2, 1 / 4, and 1 / 8 scale feature maps of the two branches are passed through three bidirectional cross attentions in parallel to obtain the cross attention feature maps of the three scales, which are then sent to the adaptive spatial feature fusion module to obtain the spatial fusion feature maps of the three scales; the bidirectional cross attention consists of parallel cross attentions. The first cross attention uses the current scale feature map of the first branch as the Value and Key, and the current scale feature map of the second branch as the Query to obtain the first branch cross attention feature map; the second cross attention uses the feature map of the second branch as the Value and Key, and the first branch feature map as the Query to obtain the second branch cross attention feature map; the first branch cross attention map and the second branch cross attention map are channel-superimposed to obtain the current scale cross attention feature map;

[0020] The prediction head includes bilinear interpolation, convolution layer, BN layer, and ReLU activation function connected in sequence to obtain the coordinate offset of the current scale;

[0021] The remapping module corrects the pixel coordinates of the PCB image to be detected by the pixel coordinate offset, and then resamples the PCB image to be detected according to the corrected coordinates to obtain an aligned PCB image;

[0022] Step 8: Crop the aligned PCB image according to the position of the component in the template image to obtain the component to be detected;

[0023] Step 9: Construct a pseudo-twin hybrid extraction network to extract the features of the components to be detected and obtain the features of the components to be detected;

[0024] The input of the pseudo-twin hybrid extraction network is the component in the template image and the component to be detected;

[0025] The pseudo-twin hybrid extraction network includes two parallel branches with shared weights. The input of the first branch is the component in the template image, and the input of the second branch is the component to be detected. The first branch is only used in the training stage, and only the second branch is used to extract the features of the component to be detected in the online detection stage. In each branch, the CNN backbone network, the Transformer encoder module, the feature dimension reduction module, and the category prediction module are connected in sequence. The CNN backbone network is composed of a CNN-based feature extraction network, the feature dimension reduction module is composed of a fully connected network, and the category prediction module is composed of a fully connected network. The output of the pseudo-twin hybrid extraction network is the feature of the component to be detected, and the predicted category output by the category prediction module is only used in the training stage.

[0026] Step 10: Construct a multi-level similarity judgment mechanism to achieve matching between the features of the component to be detected and the features of the component in the template image, and obtain the component matching result;

[0027] The construction of a multi-level similarity judgment mechanism specifically includes the following steps:

[0028] Step 10-1: Calculate the cosine similarity between the features of the component to be detected and the features of the component in the template image, and perform the first direct similarity matching;

[0029] Step 10-2: Determine whether the cosine similarity is greater than the similarity threshold of component matching. If it is greater than the threshold, the component matching is successful and go to step 10-6; otherwise, go to step 10-3;

[0030] Step 10-3: Calculate the cosine similarity between the unmatched components and all the matched components, and perform a second local association matching;

[0031] Step 10-4: Determine whether the maximum value of all cosine similarities is greater than the similarity threshold of component matching. If it is greater than the threshold, the component matching is successful and go to step 10-6; otherwise, go to step 10-5;

[0032] Step 10-5: Find the component in the template image that has the maximum cosine similarity with the component that failed to be matched, and complete the final global similarity optimization matching;

[0033] Step 10-6: assigning the category corresponding to the component in the template image to the successfully matched component;

[0034] Obtaining component matching results includes determining the type of component to be detected, if the match is successful, outputting normal, if the match is unsuccessful and it is background, outputting missed soldering, if the match is unsuccessful and it is not background, outputting wrong soldering;

[0035] Step 11: If the match is successful, go to step 2, otherwise go to step 12;

[0036] Step 12: Display the components that failed to match in the PCB image, and provide prompts for missed soldering and wrong soldering.

[0037] Furthermore, the system initialization specifically includes the following steps:

[0038] Step 1-1: Load the template image;

[0039] Step 1-2: Load the position, category and features of the components in the template image;

[0040] Step 1-3: Set the recognition area in the shooting picture;

[0041] Step 1-4: Set the integrity threshold for the PCB image to enter the camera acquisition screen;

[0042] Step 1-5: Set the similarity threshold for component matching.

[0043] Furthermore, the detection of the completeness of the PCB image in the current video frame entering the camera acquisition screen is achieved through an image classification network. The video frame is input into the image classification network to obtain complete and incomplete classification probability values ​​of the PCB image in the video frame; if the complete classification probability value is greater than the completeness threshold, the completeness result is complete, otherwise it is incomplete.

[0044] Furthermore, the screening and processing of the collected multiple frames of PCB images in step 6 specifically includes the following steps:

[0045] Step 6-1: Use the semantic segmentation network to obtain the background segmentation map and center coordinate heat map of multiple frames of PCB images;

[0046] Step 6-2: Obtain the center coordinates of multiple frames of PCB images according to the center coordinate heat map;

[0047] Step 6-3: Determine the video frame with the smallest distance between the center of the camera capture screen and the center coordinates as the PCB image to be detected;

[0048] Step 6-4: According to the background segmentation map of the PCB image to be detected, the background area pixels in the PCB image to be detected are set to 0.

[0049] A detection device for a detection method of a PCB surface component, comprising:

[0050] Camera module: used to capture high-resolution circuit board images in real time;

[0051] Light source module: used to provide sufficient light;

[0052] Conveying module: used to continuously convey the PCB to be inspected to the camera module collection location in a fixed direction;

[0053] Alignment module: contains a multi-scale twin spatial deformation perception network, which is used for spatial alignment of the PCB image to be inspected with the template image;

[0054] Extraction module: contains a pseudo-twin hybrid extraction network to extract features of the components to be detected;

[0055] Matching module: contains a multi-level similarity judgment mechanism to achieve matching between the features of the component to be detected and the features of the component in the template image;

[0056] Display module: used to display the PCB image, template image and detection results collected by the camera module in real time;

[0057] Database module: used to store template image information, system parameters and network model parameters;

[0058] Computer: used to run all programs, receive and process real-time data transmitted by the camera module;

[0059] The computer is connected to the camera module and the display module;

[0060] The computer runs a database module, an alignment module, an extraction module and a matching module.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1) Through the multi-scale twin space deformation perception network, pseudo-twin hybrid extraction network and multi-level similarity judgment mechanism, the PCB image to be inspected is accurately aligned and matched with the template image of the current batch, so as to realize the accurate detection of missed soldering and wrong soldering of electronic components on PCB and avoid the problem of missed detection and wrong detection of small targets;

[0063] 2) A multi-scale twin spatial deformation perception network is adopted to effectively reduce the initial deformation difference between the template image and the image to be detected through the pre-alignment network, providing better input for subsequent fine alignment. Through multi-scale feature extraction and spatial deformation perception mechanism, global correlation modeling and fine capture of local details are realized, and sufficient information interaction is carried out in different directions, effectively eliminating the spatial deformation between the template image and the PCB image to be detected. In addition, multi-scale information is fully utilized through autonomous adjustment of fusion weights and parallel prediction, thereby improving the accuracy of coordinate offset prediction;

[0064] 3) A pseudo-twin hybrid extraction network is used to achieve comparative feature learning between the PCB image to be tested and the template image through the pseudo-twin structure during the training phase, making the model more robust when dealing with different component shapes, angles or lighting changes. It also simplifies the calculation in the online phase, reduces the demand for hardware resources, and improves real-time performance. This network structure can achieve efficient capture of local features and global correlation information, enhance the sensitivity of features to category differences, enrich feature expression, and reduce computational complexity.

[0065] 4) A multi-level similarity judgment mechanism is provided for PCB component matching anomalies. Direct similarity matching quickly locks the matching relationship and avoids complex calculations when the feature extraction quality is high; local association matching uses the similarity information of successfully matched components to compensate for feature deviations caused by factors such as lighting and angle, and corrects omissions in direct similarity matching; global similarity optimization globally analyzes the features of unmatched components and all components in the template image to ensure that potential correct matches are not missed, and to ensure the reliability of the matching mechanism in extreme cases; this multi-level similarity judgment mechanism performs three matches from coarse to fine, from local to global, step by step, to ensure both efficiency and accuracy, and through step-by-step optimization and multiple matching judgments, effectively reduces mismatches caused by external interference and improves the reliability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of the method described in an embodiment of the present invention.

[0067] Figure 2 It is a flowchart of system initialization described in an embodiment of the present invention.

[0068] Figure 3 It is a flow chart of screening and processing the collected multiple frames of PCB images according to an embodiment of the present invention.

[0069] Figure 4 It is a flow chart of constructing a multi-level similarity judgment mechanism according to an embodiment of the present invention.

[0070] Figure 5 It is a structural block diagram of the multi-scale twin space deformation perception network described in an embodiment of the present invention.

[0071] Figure 6 It is a structural block diagram of the spatial deformation perception module described in an embodiment of the present invention.

[0072] Figure 7 It is a structural block diagram of the pseudo-twin hybrid extraction network described in an embodiment of the present invention.

[0073] Figure 8 It is a schematic diagram of the structure of the detection device for PCB surface components according to an embodiment of the present invention.

[0074] Fig. 9 It is a schematic diagram of the application of the detection device for PCB surface components according to an embodiment of the present invention.

[0075] In the figure: 1. Camera 2. Light source 3. Belt conveyor 4. Display 5. Computer DETAILED DESCRIPTION

[0076] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0077] See Figure 1 , is a flow chart of the method of the present invention. A method for detecting components on a PCB surface of the present invention comprises the following steps:

[0078] S1: System initialization: System initialization completes the parameter settings and file loading required for system operation; see Figure 2 , the system initialization specifically includes the following steps:

[0079] S1.1: Load template image;

[0080] The template image is the standard PCB image of the current inspection batch, which is used to provide a standard PCB image style;

[0081] S1.2: Load the position, category and features of the components in the template image;

[0082] Provides relative position information and category information of components in standard PCB images, as well as pre-extracted feature information of PCB component images;

[0083] S1.3: Set the recognition area in the shooting picture;

[0084] In order to reduce image deformation caused by excessive shooting angles in the image of the PCB to be inspected, the effective recognition area in the picture is set according to the resolution captured by the camera and the shape and size of the PCB, and the video frame needs to be cropped in real time according to the set recognition area.

[0085] S1.4: Set the completeness threshold of the PCB image entering the camera acquisition screen;

[0086] Set the threshold for PCB integrity detection and obtain the integrity result;

[0087] S1.5: Set the similarity threshold for component matching;

[0088] A similarity threshold for component matching in a multi-level similarity judgment mechanism is set to obtain component matching results.

[0089] S2.0: Capture video frames through the camera; obtain the video frames captured by the camera, and crop the video frames of the camera in real time according to the recognition area in the shooting picture set when the system is initialized.

[0090] S3.0: Detect the completeness of the PCB image in the current video frame entering the camera acquisition screen to obtain the completeness result;

[0091] The completeness detection is implemented using the ResNet-18 classification network. The input image resolution of the ResNet-18 network is 224×224. The video frame is input into the ResNet-18 classification network to obtain the complete and incomplete classification probability values ​​of the PCB image in the video frame.

[0092] If the complete classification probability value is greater than the completeness threshold, the completeness result is complete, otherwise it is incomplete;

[0093] The offline training process of the ResNet-18 network is as follows:

[0094] The training data set uses the public PCB data set, which is randomly cropped to form an incomplete PCB image, and is used together with the original complete image as the training data set. The optimizer uses the stochastic gradient descent method SGD, the loss function uses the cross entropy loss, and the initial learning rate is 10 -3, the batch size is 64 and the number of iterations is 100;

[0095] The images in the training set are fed into the recognition model at once, the ResNet-18 network is iteratively trained, and cross-validated with the validation set to obtain the optimal weight parameters and hyperparameters of the ResNet-18 network;

[0096] In order to prevent misdetection by the ResNet-18 classification network, the completeness results of 10 consecutive frames are used as the final judgment basis. If the PCB images in 10 consecutive video frames are judged to be complete, the PCB image in the video frame is considered to be complete.

[0097] S4.0: Determine the integrity result; determine the subsequent processing steps based on the PCB integrity result obtained by the integrity test. If the PCB integrity result is complete, it is necessary to continue the subsequent processing flow. If it is incomplete, it is necessary to continue to obtain the camera's video frame and go to step S2.0 to repeat the above process.

[0098] S5.0: Continuously acquire multiple frames of images; detect the frame images acquired by the cache camera at fixed time intervals until 10 consecutive frames of integrity results are obtained, and then stop acquiring.

[0099] S6.0: Screen and process the collected multiple frames of PCB images to obtain the PCB image to be detected; screen out the frame with the PCB closest to the center of the image in the cached multiple frames, and set the background pixels of the screened frame to 0, see Figure 2 , screening and processing of PCB images include the following steps:

[0100] S6.1: Use the semantic segmentation network to obtain the background segmentation map and center coordinate heat map of multiple frames of PCB images;

[0101] The semantic segmentation network is implemented using the U-Net network, as described below:

[0102] The input of the semantic segmentation network using the U-Net network is the collected multi-frame PCB images, with 1 channel and a resolution of 224×224. The output is the background segmentation map and the center coordinate heat map, with 2 channels and a resolution of 224×224.

[0103] The offline training process of the image segmentation model U-Net network is as follows:

[0104] The training dataset is annotated from the public PCB dataset. The optimizer uses the stochastic gradient descent method SGD, the segmentation map loss function uses the cross entropy loss, the prediction center heat map loss function uses the mean square error loss, and the initial learning rate is 10 -3 , the batch size is 16 and the number of iterations is 100;

[0105] The images in the training set are sequentially sent to the image segmentation model, the U-Net network of the image segmentation model is iteratively trained, and cross-validated through the validation set to obtain the optimal weight parameters and hyperparameters of the U-Net network of the image segmentation model;

[0106] S6.2: Obtain the center coordinates of multiple frames of PCB images according to the center coordinate heat map;

[0107] According to the maximum response position of the center coordinate heat map, the center coordinate of the PCB image is used;

[0108] S6.3: Determine the video frame with the smallest distance between the center of the camera capture screen and the center coordinates as the PCB image to be detected;

[0109] Use the Euclidean distance to calculate the distance between the center of the camera capture image and the center coordinate of the PCB image. The calculation formula is as follows:

[0110]

[0111] Among them, x and y are the center coordinates of the PCB image, and w and h are the width and height of the acquisition screen;

[0112] The frame with the smallest distance is used as the PCB image to be detected;

[0113] S6.4: according to the background segmentation map of the PCB image to be detected, setting the background area pixels in the PCB image to be detected to 0;

[0114] According to the background segmentation map of the PCB image to be detected, the pixel values ​​of the part of the PCB image to be detected corresponding to the background segmentation map less than 0.5 are set to 0, thereby shielding the background of the PCB image to be detected.

[0115] S7.0: Construct a multi-scale twin space deformation perception network to achieve spatial alignment between the PCB image to be detected and the template image, and obtain the aligned PCB image; the construction of the multi-scale twin space deformation perception network includes:

[0116] See Figure 5The multi-scale twin spatial deformation perception network consists of two parallel branches. The input of the first branch is the template image. The second branch inputs the template image and the PCB image to be detected into the pre-alignment network to generate a pre-aligned PCB image as the input of the second branch; the two branches successively pass through the feature extraction module with shared weights to extract the feature maps of the input image resolution at scales of 1 / 2, 1 / 4, and 1 / 8 respectively; the feature maps of the three scales of the two branches are simultaneously input into the spatial deformation perception module to generate the fused three-scale feature maps; the fused three-scale feature maps are respectively input into the three parallel prediction heads, and the prediction results of the three prediction heads are added to obtain the pixel coordinate offset; the remapping module uses the pixel coordinate offset to transform the PCB image to be detected to obtain the aligned PCB image;

[0117] The pre-alignment network uses an improved STN network. The improved STN network replaces the two 3×3 convolutions in the Basic Block of the positioning network Dailated ResNet-18 with a set of 3×3 dilated convolutions with dilation rates of 1, 3, and 5. The dilated convolution can increase the receptive field of the network and enable the network to perceive a larger spatial offset.

[0118] The feature extraction module uses the ENET network, which outputs the encoder stage feature layer as a small-scale feature, the first upsampling layer as a medium-scale feature, and the second upsampling layer as a large-scale feature. Through multi-scale feature extraction, spatial features of different scales can be obtained, thereby improving the accuracy of alignment.

[0119] See Figure 6 In the spatial deformation perception module, the 1 / 2, 1 / 4, and 1 / 8 scale feature maps of the two branches are parallelly passed through three bidirectional cross attentions to obtain the cross attention feature maps of the three scales, and then sent to the adaptive spatial feature fusion module to obtain the spatial fusion feature maps of the three scales; the bidirectional cross attention consists of parallel cross attentions. The first cross attention uses the current scale feature map of the first branch as Value and Key, and the current scale feature map of the second branch as Query to obtain the first branch cross attention feature map; the second cross attention uses the feature map of the second branch as Value and Key, and the first branch feature map as Query to obtain the second branch cross attention feature map; the first branch cross attention map and the second branch cross attention map are channel-superimposed to obtain the current scale cross attention feature map; through cross attention, the network can learn the spatial contrast features between images, and the adaptive spatial features can adaptively learn the weights of features in different scales. Through spatial deformation perception, the network can fuse multi-scale spatial features from two images, thereby learning the importance of spatial contrast features at different scales;

[0120] The structure of the prediction head is shown in Table 1, which includes bilinear interpolation, convolution layer, BN layer, and ReLU activation function connected in sequence to obtain the coordinate offset of the current scale. The parallel prediction head can automatically allocate the contribution of different scales to the pixel offset in the image.

[0121] Table 1

[0122] type Input Channels Output Channel Bilinear interpolation 128 / 64 / 32 128 / 64 / 32 Convolution 3×3, padding 1 128 / 64 / 32 16 Batch Normalization 16 16 ReLU activation function 16 16 Convolution 3×3, padding 1 16 8 Batch Normalization 8 8 ReLU activation function 8 16 Convolution 1×1 8 2

[0123] The remapping module corrects the pixel coordinates of the PCB image to be detected by the pixel coordinate offset, and then resamples the PCB image to be detected according to the corrected coordinates to obtain an aligned PCB image;

[0124] The pixel coordinate offset is the offset of the network input image resolution. It is necessary to correct the pixel coordinate offset according to the PCB image to be detected. The coordinate calculation formula of the pixel after correction is as follows:

[0125] C=C 0 +H scale ·C offset (2)

[0126] Among them, C represents the corrected coordinates, C 0 is the original coordinate, H scale C is the scaling matrix from the PCB image resolution to be inspected to the network input image resolution, offset is the pixel coordinate offset; the final aligned PCB image is obtained by resampling the pixels to new coordinates and then smoothing them by bilinear interpolation;

[0127] The offline training process of the multi-scale twin space deformation perception network is as follows:

[0128] The training set uses the CoCo image dataset, the optimizer uses AdamW, the loss function uses SmoothL1 loss, the number of iterations is 500, the batch size is 32, and the learning rate is 10 -4 ;

[0129] The images in the training set are randomly transformed and compared with the original Figure 1 The multi-scale twin space deformation perception network is iteratively trained by inputting the model, and cross-validated through the validation set to obtain the optimal weight parameters and hyperparameters of the multi-scale twin space deformation perception network.

[0130] S8.0: Crop the aligned PCB image according to the position of the component in the template image to obtain the component to be detected;

[0131] All components in the aligned PCB image are cropped using the positions of the components in the template image read during system initialization, thereby obtaining all the components to be detected in the aligned PCB image.

[0132] S9.0: Construct a pseudo-twin hybrid extraction network to extract the features of the components to be detected and obtain the features of the components to be detected;

[0133] Construct a pseudo-twin hybrid extraction network, which includes the following contents:

[0134] See Figure 7 ,The input of the pseudo-twin hybrid extraction network is the component in the template image and the component to be detected;

[0135] The pseudo-twin hybrid extraction network includes two parallel branches with shared weights. The input of the first branch is the component in the template image, and the input of the second branch is the component to be detected. The first branch is only used in the training stage, and only the second branch is used to extract the features of the component to be detected in the online detection stage. In each branch, the CNN backbone network, the Transformer encoder module, the feature dimension reduction module, and the category prediction module are connected in sequence. The CNN backbone network is implemented using the pre-trained Resnet-50, and the Transformer encoder module is the Transformer Encoder. The embedding dimension of the input and output is 512. The component features are extracted by the CNN backbone network and the Transformer encoder. The advantages of CNN and Transformer can be combined to capture the local and global features of the component to be detected, ensuring the richness of feature expression.

[0136] The feature dimension reduction module consists of a three-layer fully connected network, which reduces the output feature dimension to 128 dimensions. The feature dimension reduction module reduces the dimension of high-dimensional features, removes redundant information, reduces computational complexity, and retains key information. The category prediction module consists of a two-layer fully connected network, predicts category information, uses category labels for supervised learning, and enhances the sensitivity of features to category differences. The output of the pseudo-twin hybrid extraction network is the component feature to be detected. The predicted category output by the category prediction module is only used in the training phase;

[0137] The offline training process of the pseudo-twin hybrid extraction network is as follows:

[0138] The training set uses 28 categories of circuit component images. During training, two components of the same type are randomly selected from all categories and sent to the network. The optimizer uses AdamW, the number of iterations is 100, the batch size is 64, and the learning rate is 10. -4The loss function uses triple loss. When calculating the loss, the same component features are used as Anchor and Positive. In addition, according to the cosine similarity of the features, the different components with the largest similarity in the batch are selected as Negative to calculate the loss. The category loss is used as an auxiliary loss using BCEWithLogitsLoss.

[0139] The images in the training set are sent into the model, the pseudo-twin hybrid extraction network is iteratively trained, and cross-validated through the validation set to obtain the optimal weight parameters and hyperparameters of the pseudo-twin hybrid extraction network.

[0140] S10.0: Construct a multi-level similarity judgment mechanism to match the features of the component to be detected with the features of the component in the template image, and obtain the matching result of the component;

[0141] S1.10.1: Calculate the cosine similarity between the features of the component to be detected and the features of the component in the template image, and perform the first direct similarity matching;

[0142] The cosine similarity between the features of the component to be detected and the features of the component in the template image is calculated using the following formula:

[0143]

[0144] Where v d is the characteristic vector of the component to be detected, v t is the feature vector of the template original, n represents the vector dimension, and i is the vector dimension index;

[0145] S10.2: Determine whether the cosine similarity is greater than the similarity threshold of component matching. If it is greater than the threshold, the component matching is successful and go to step S10.6. Otherwise, go to step S10.3. The components to be detected in the PCB and the components in the template image are components of the same appearance and the same model. These components are directly matched by similarity, and components with higher matching probability can be matched first, so that the matching relationship can be quickly locked.

[0146] S10.3: Calculate the cosine similarity between the components that failed to match and all the components that successfully matched, and perform a second local association matching;

[0147] S10.4: Determine whether the maximum value of all cosine similarities is greater than the similarity threshold of component matching. If it is greater than the threshold, the component matching is successful and go to step S10.6. Otherwise, go to step S10.5. The components that are successfully matched directly have a higher probability of being correctly matched, and the difference of the same component image in the same PCB will be smaller. The components that fail to be directly matched are locally associated and matched, which can better find the best match for the components that fail to be directly matched and correct the omissions in the direct similar matching.

[0148] S10.5: Find the component in the template image that has the maximum cosine similarity with the component that failed to be matched, and complete the final global similarity optimization matching; if both direct similarity matching and local association matching fail, perform global similarity optimization matching with the component in the template image to prevent the matching from failing to reach the similarity threshold due to shooting reasons such as lighting and angle; the template image is usually a high-quality image that has been optimized and annotated, and its component features have high recognition. This global similarity optimization matching can ensure the reliability of the matching mechanism in extreme cases;

[0149] S10.6: assigning the category corresponding to the component in the template image to the successfully matched component;

[0150] The category corresponding to the component in the template image is assigned to the successfully matched component.

[0151] S11.0: If the match is successful, go to S2.0, otherwise go to S12.0;

[0152] If the match is successful, the current PCB detection passes, otherwise a corresponding prompt is required.

[0153] S12.0: Displays the components that failed to match in the PCB image, and provides prompts for missed soldering and wrong soldering.

[0154] The position of the components that failed to match in the PCB image is displayed, and the missed soldering and wrong soldering of the components can be determined by comparing with the categories of the components in the template image, and prompts for missed soldering and wrong soldering can be given.

[0155] See Figure 8 A detection device for PCB surface components includes a camera module, a light source module, a transmission module, an alignment module, an extraction module, a matching module, a display module, a database module, and a computer. The alignment module, the extraction module, the matching module, and the database module are run in the computer, and the display module and the camera module are connected to the computer.

[0156] See Fig. 9 The conveyor module adopts a belt transmission mechanism, the camera module adopts the MV-2000GC camera manufactured by Vision Intelligent Manufacturing, the light source module adopts the AFT-RL12068W ring light source manufactured by Vision Intelligent Manufacturing, the display adopts a 24-inch LCD display, the camera module is connected to the computer via an RJ45 network, and the display is connected to the computer via an HDMI interface;

[0157] Camera module: It consists of the MV-2000GC camera 1 manufactured by Vision Intelligent Manufacturing, which is used to collect the PCB image of the conveyor moving in real time;

[0158] Light source module: It is composed of AFT-RL12068W ring light source 2 manufactured by Vision Intelligent Manufacturing, which is used to compensate the light of camera 1 to ensure that the camera module can capture clear images;

[0159] Conveying module: It is composed of a belt conveying mechanism 3, used to carry the PCB and move it in a straight line to the bottom of the camera;

[0160] Alignment module: Contains a multi-scale twin spatial deformation perception network, which is used for spatial alignment of the PCB image to be inspected and the template image, including:

[0161] Offline training of the multi-scale twin space deformation perception network to obtain the optimal weight parameters and hyperparameters;

[0162] Extraction module: Contains a pseudo-twin hybrid extraction network to extract features of the components to be detected, including:

[0163] Offline training of the pseudo-twin hybrid extraction network to obtain the optimal weight parameters and hyperparameters;

[0164] Matching module: contains a multi-level similarity judgment mechanism to achieve matching between the features of the component to be detected and the features of the component in the template image;

[0165] Database module: uses a solid-state storage hard disk to save data and parameters of PCB surface component detection methods.

[0166] Display module: Display 4 is used for system interface, real-time display of video frames collected by the camera module and real-time processing results of the alignment module and matching module, including:

[0167] Display system start, stop, configuration and other interactive components;

[0168] Display the video frames acquired by the camera;

[0169] Display the PCB image after alignment by the alignment module;

[0170] Display the position of the component that failed to match the detected component in the matching module;

[0171] Display the leaking and wrong welding conditions in the components to be tested.

[0172] Computer 5: used to drive the camera module and run the PCB surface component detection method program, including:

[0173] Drive the MV-2000GC camera 1 produced by Vision Intelligent Manufacturing;

[0174] Run the database module to save the data and parameters of the PCB surface component detection method;

[0175] Run the alignment module to complete the spatial alignment between the PCB image to be inspected and the template image;

[0176] Running the extraction module to complete feature extraction of the component to be detected;

[0177] Running the matching module to complete the matching between the features of the component to be detected and the features of the component in the template image;

[0178] Provide GPU computing power support for the alignment module, extraction module, and matching module.

[0179] The above embodiments are implemented based on the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the above embodiments. The methods used in the above embodiments are conventional methods unless otherwise specified.

Claims

1. A method for detecting components on a PCB surface, characterized in that: The steps include: Step 1: System initialization; Step 2: Capture video frames through the camera; Step 3: Detect the completeness of the PCB image in the current video frame entering the camera acquisition screen to obtain the completeness result; Step 4: If the completeness result is incomplete, go to step 2, otherwise go to step 5; Step 5: Continuously collect multiple frames of PCB images at fixed time intervals; Step 6: Screen and process the collected multiple frames of PCB images to obtain the PCB image to be tested; Step 7: Construct a multi-scale twin spatial deformation perception network to achieve spatial alignment between the PCB image to be detected and the template image, and obtain the aligned PCB image; The input of the multi-scale twin space deformation perception network is the template image and the PCB image to be detected; The multi-scale twin spatial deformation perception network consists of two parallel branches. The input of the first branch is the template image. The second branch inputs the template image and the PCB image to be detected into the pre-alignment network to generate a pre-aligned PCB image as the input of the second branch. The two branches successively pass through the multi-scale feature extraction module with shared weights to extract feature maps at scales of 1 / 2, 1 / 4, and 1 / 8 of the input image resolution respectively. The feature maps of the three scales of the two branches are simultaneously input into the spatial deformation perception module to generate fused feature maps of three scales. The fused feature maps of three scales are respectively input into three parallel prediction heads, and the prediction results of the three prediction heads are added to obtain the pixel coordinate offset. The remapping module uses the pixel coordinate offset to transform the PCB image to be detected to obtain the aligned PCB image. The pre-aligned network uses an improved STN network, which replaces the ordinary convolution in the STN network with dilated convolution; The feature extraction module uses the ENET network, which outputs the encoder stage feature layer as small-scale features, the first upsampling layer as medium-scale features, and the second upsampling layer as large-scale features. In the spatial deformation perception module, the 1 / 2, 1 / 4, and 1 / 8 scale feature maps of the two branches are passed through three bidirectional cross attentions in parallel to obtain the cross attention feature maps of the three scales, which are then sent to the adaptive spatial feature fusion module to obtain the spatial fusion feature maps of the three scales; the bidirectional cross attention consists of parallel cross attentions. The first cross attention uses the current scale feature map of the first branch as the Value and Key, and the current scale feature map of the second branch as the Query to obtain the first branch cross attention feature map; the second cross attention uses the feature map of the second branch as the Value and Key, and the first branch feature map as the Query to obtain the second branch cross attention feature map; the first branch cross attention map and the second branch cross attention map are channel-superimposed to obtain the current scale cross attention feature map; The prediction head includes bilinear interpolation, convolution layer, BN layer, and ReLU activation function connected in sequence to obtain the coordinate offset of the current scale; The remapping module corrects the pixel coordinates of the PCB image to be detected by the pixel coordinate offset, and then resamples the PCB image to be detected according to the corrected coordinates to obtain an aligned PCB image; Step 8: Crop the aligned PCB image according to the position of the component in the template image to obtain the component to be detected; Step 9: Construct a pseudo-twin hybrid extraction network to extract the features of the components to be detected and obtain the features of the components to be detected; The input of the pseudo-twin hybrid extraction network is the component in the template image and the component to be detected; The pseudo-twin hybrid extraction network includes two parallel branches with shared weights. The input of the first branch is the component in the template image, and the input of the second branch is the component to be detected. The first branch is only used in the training stage, and only the second branch is used to extract the features of the component to be detected in the online detection stage. In each branch, the CNN backbone network, the Transformer encoder module, the feature dimension reduction module, and the category prediction module are connected in sequence. The CNN backbone network is composed of a CNN-based feature extraction network, the feature dimension reduction module is composed of a fully connected network, and the category prediction module is composed of a fully connected network. The output of the pseudo-twin hybrid extraction network is the feature of the component to be detected, and the predicted category output by the category prediction module is only used in the training stage. Step 10: Construct a multi-level similarity judgment mechanism to achieve matching between the features of the component to be detected and the features of the component in the template image, and obtain the component matching result; The construction of a multi-level similarity judgment mechanism specifically includes the following steps: Step 10-1: Calculate the cosine similarity between the features of the component to be detected and the features of the component in the template image, and perform the first direct similarity matching; Step 10-2: Determine whether the cosine similarity is greater than the similarity threshold of component matching. If it is greater than the threshold, the component matching is successful and go to step 10-6; otherwise, go to step 10-3; Step 10-3: Calculate the cosine similarity between the unmatched components and all the matched components, and perform a second local association matching; Step 10-4: Determine whether the maximum value of all cosine similarities is greater than the similarity threshold of component matching. If it is greater than the threshold, the component matching is successful and go to step 10-6; otherwise, go to step 10-5; Step 10-5: Find the component in the template image that has the maximum cosine similarity with the component that failed to be matched, and complete the final global similarity optimization matching; Step 10-6: assigning the category corresponding to the component in the template image to the successfully matched component; Obtaining component matching results includes determining the type of component to be detected, if the match is successful, outputting normal, if the match is unsuccessful and it is background, outputting missed soldering, if the match is unsuccessful and it is not background, outputting wrong soldering; Step 11: If the match is successful, go to step 2, otherwise go to step 12; Step 12: Display the components that failed to match in the PCB image, and provide prompts for missed soldering and wrong soldering.

2. A method for detecting components on a PCB surface according to claim 1, characterized in that: The system initialization specifically includes the following steps: Step 1-1: Load the template image; Step 1-2: Load the position, category and features of the components in the template image; Step 1-3: Set the recognition area in the shooting picture; Step 1-4: Set the integrity threshold for the PCB image to enter the camera acquisition screen; Step 1-5: Set the similarity threshold for component matching.

3. A method for detecting components on a PCB surface according to claim 1, characterized in that: The detection of the completeness of the PCB image in the current video frame entering the camera acquisition screen is achieved through an image classification network. The video frame is input into the image classification network to obtain complete and incomplete classification probability values ​​of the PCB image in the video frame; if the complete classification probability value is greater than the completeness threshold, the completeness result is complete, otherwise it is incomplete.

4. A method for detecting components on a PCB surface according to claim 1, characterized in that: The screening and processing of the collected multiple frames of PCB images in step 6 specifically includes the following steps: Step 6-1: Use the semantic segmentation network to obtain the background segmentation map and center coordinate heat map of multiple frames of PCB images; Step 6-2: Obtain the center coordinates of multiple frames of PCB images according to the center coordinate heat map; Step 6-3: Determine the video frame with the smallest distance between the center of the camera capture screen and the center coordinates as the PCB image to be detected; Step 6-4: According to the background segmentation map of the PCB image to be detected, the background area pixels in the PCB image to be detected are set to 0.

5. A detection device for a method for detecting components on a PCB surface according to any one of claims 1 to 4, characterized in that: include: Camera module: used to capture high-resolution circuit board images in real time; Light source module: used to provide sufficient light; Conveying module: used to continuously convey the PCB to be inspected to the camera module collection location in a fixed direction; Alignment module: contains a multi-scale twin spatial deformation perception network, which is used for spatial alignment of the PCB image to be inspected with the template image; Extraction module: contains a pseudo-twin hybrid extraction network to extract features of the components to be detected; Matching module: contains a multi-level similarity judgment mechanism to achieve matching between the features of the component to be detected and the features of the component in the template image; Display module: used to display the PCB image, template image and detection results collected by the camera module in real time; Database module: used to store template image information, system parameters and network model parameters; Computer: used to run all programs, receive and process real-time data transmitted by the camera module; The computer is connected to the camera module and the display module; The computer runs a database module, an alignment module, an extraction module and a matching module.

Images