A PCBA solder joint defect detection method and system based on improved RT-DETR

Abstract

The present invention provides a PCBA solder joint defect detection method and system based on improved RT-DETR in the field of industrial AI quality inspection technology. The method includes: step S1, acquiring a large number of historical PCBA solder joint images to construct a dataset; step S2, creating a solder joint defect detection model based on a feature extraction layer, a feature fusion layer, and a prediction output layer; step S3, dividing the dataset into a training set, a validation set, and a test set, training the solder joint defect detection model with the training set, validating the trained solder joint defect detection model with the validation set, and testing the validated solder joint defect detection model with the test set; step S4, compressing, drift-compensating, and deploying the tested solder joint defect detection model; and step S5, performing PCBA solder joint defect detection using the deployed solder joint defect detection model. The present invention has the advantage of greatly improving the accuracy of PCBA solder joint defect detection while meeting real-time detection requirements.

Description

Technical Field

[0001] The present invention relates to the field of industrial AI quality inspection technology, and in particular to a PCBA solder joint defect detection method and system based on improved RT-DETR. Background Art

[0002] With the widespread adoption of electronic products, the electronics assembly industry continues to grow rapidly. As a core component of electronic products, the reliable inspection of solder joint quality in printed circuit board assemblies (PCBAs) is crucial. Currently, manually assisted solder joint quality inspection equipment is widely used within the industry, particularly for solder joint inspection of non-standard electronic components, which relies heavily on manual visual inspection. This largely manual inspection method suffers from disadvantages such as low efficiency, high labor costs, and high subjectivity. To improve inspection efficiency and objectivity, automated inspection technologies based on image recognition are gaining increasing attention.

[0003] In industrial applications, images for solder joint quality inspection are primarily captured using industrial cameras. Compared to the clear outlines and simple backgrounds typically seen in general target detection, these images present significant challenges: The targets (such as solder joints and their defects) are typically small, the background information is complex, the targets are densely distributed, the defect categories are diverse, and some feature differences between categories are subtle. These factors combine to make accurate detection and detailed classification of solder joint defects challenging.

[0004] In recent years, deep learning-based computer vision technology has been widely used in the field of industrial defect detection. Mainstream deep learning detection methods mainly include two categories: those based on convolutional neural networks (CNNs) and those based on the Transformer architecture. CNNs have advantages in extracting local features in image space, but they are limited by their local receptive field. When dealing with small objects or complex background interference, their ability to model global information is limited, which can easily lead to false detections or missed detections. As a result, their performance in PCBA solder joint defect detection tasks is less than ideal. Transformer-based object detection methods, due to their global modeling capabilities and self-attention mechanism, have shown great potential for handling complex inspection tasks. However, the high resource consumption caused by calculating attention scores makes meeting the real-time requirements of inspection tasks a challenge.

[0005] To overcome the computational bottleneck of the Transformer model, the RT-DETR model was developed. RT-DETR significantly reduces the resource overhead caused by calculating attention scores through architectural optimization, and can meet the real-time requirements of solder joint defect detection, making it have application prospects in the field of industrial inspection.Figure 5 As shown in Figure 2, RT-DETR includes a key component, RepC3, which is mainly used in the feature fusion stage to achieve effective fusion of shallow feature maps (rich in detail information) and deep feature maps (rich in semantic information). Specifically, the RepC3 module includes:

[0006] Input: used to receive input features x;

[0007] First branch: After performing a 1×1 convolution on the input feature x, it is processed by at least one repeating module. Repeating modules can be stacked N times as needed. Each repeating module contains sub-branch a and sub-branch b for feature processing. The features processed by sub-branch a and sub-branch b are element-wise added together and then processed through the SiLU activation function. The result serves as the input of the next repeating module or the final output of the first branch.

[0008] Second branch: perform 1×1 convolution operation on the input feature x;

[0009] Output end: The output feature map of the first branch is added to the output feature map of the second branch element by element to generate the final fused output feature map.

[0010] However, when the aforementioned RepC3 structure is applied to the task of inspecting small-scale solder joints in the complex context of PCBAs, it suffers from insufficient multi-scale information fusion capabilities. The core of the problem lies in the fact that the same type of target (such as a solder joint or a specific defect) appears as an abstract semantic region in deep high-level feature maps, but as irregularly shaped pixel blocks in shallow low-level feature maps, resulting in a significant semantic gap between feature maps at different levels. RepC3 primarily relies on convolution operations for cross-layer information exchange. Due to the inherent receptive field limitations of standard convolution operations, it is difficult to bridge the huge differences between these different semantic spaces and establish effective correspondences between features. This results in unsatisfactory feature fusion effects, and key details of small targets (such as solder joints) are easily lost during the fusion process, ultimately limiting the overall detection accuracy of the model.

[0011] Therefore, how to provide a PCBA solder joint defect detection method and system based on improved RT-DETR to improve the accuracy of PCBA solder joint defect detection while meeting real-time detection requirements has become a technical problem that needs to be solved urgently. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to provide a PCBA solder joint defect detection method and system based on improved RT-DETR, so as to improve the accuracy of PCBA solder joint defect detection while meeting the real-time detection requirements.

[0013] In a first aspect, the present invention provides a PCBA solder joint defect detection method based on improved RT-DETR, comprising the following steps:

[0014] Step S1: Acquire a large number of historical PCBA solder joint images, pre-process and annotate each of the historical PCBA solder joint images, and then construct a data set;

[0015] Step S2: creating a solder joint defect detection model based on the feature extraction layer, the feature fusion layer, and the prediction output layer, and setting a loss function of the solder joint defect detection model;

[0016] Step S3, dividing the data set into a training set, a validation set, and a test set, training a solder joint defect detection model using the training set and a loss function, validating the trained solder joint defect detection model using the validation set, and testing the validated solder joint defect detection model using the test set;

[0017] Step S4: compressing, drift compensation training, and deploying the solder joint defect detection model that has passed the test;

[0018] Step S5: Perform PCBA solder point defect detection using the deployed solder point defect detection model.

[0019] Furthermore, the step S1 is specifically as follows:

[0020] A large number of historical PCBA solder joint images with different lighting conditions, shooting angles, and PCBA models are acquired. Each of the historical PCBA solder joint images is preprocessed, including at least cropping, contrast adjustment, exposure adjustment, and size adjustment. The preprocessed historical PCBA solder joint images are labeled with defect types and defect locations using the labelimg annotation tool. A dataset is constructed based on the labeled historical PCBA solder joint images. The defect types include at least normal, kneeling foot, excessive solder, insufficient solder, and continuous solder.

[0021] Furthermore, in step S2, the feature extraction layer is constructed based on the backbone network of the RT-DETR model, and is used to extract shallow features and deep features from the input PCBA solder joint image, and input the shallow features and deep features into the feature fusion layer;

[0022] The feature fusion layer is constructed based on the local feature processing module, the global feature processing module and the feature interaction output module;

[0023] The local feature processing module is constructed based on an input channel adjustment unit, a local enhancement unit, a structure optimization unit, and a residual connection unit; the input channel adjustment unit is used to perform channel compression and preliminary feature extraction on the input shallow features and deep features through a 1x1 first convolution normalization activation layer to obtain initial features; the local enhancement unit is used to enhance the local perception ability of the initial features and introduce nonlinear transformation through two stacked local enhancement sub-units to obtain enhanced initial features; the structure optimization unit is used to fuse the two-branch structures of the two local enhancement sub-units into a single 3x3 convolution layer in the inference stage; the residual connection unit is used to fuse the enhanced initial features, shallow features, and deep features to obtain local features, and input the local features into the global feature processing module;

[0024] The global feature processing module is constructed based on an attention generation unit, a channel space mapping unit, and a spatial compression unit; the attention generation unit is used to fuse the local features through global average pooling, global maximum pooling, and a Softmax function to obtain a fused attention feature; the channel space mapping unit is used to perform spatial mapping on the local features through a 1x1 second convolution normalized activation layer to obtain a channel space mapping feature; the spatial compression unit is used to perform spatial compression on the local features through a 1x1 third convolution normalized activation layer to obtain a spatial compression feature;

[0025] The feature interaction output module is used to perform matrix multiplication on the fused attention feature and the channel space mapping feature to obtain a spatial enhancement feature, perform matrix multiplication on the channel space mapping feature and the spatial compression feature to obtain a channel enhancement feature, input the spatial enhancement feature into the 1x1 fourth convolution normalized activation layer to obtain a spatial attention map, and input the channel enhancement feature into the unnormalized convolution activation layer to obtain a channel attention map; perform element-by-element multiplication on the spatial attention map and the channel attention map to obtain an initial fused feature, perform a residual connection on the initial fused feature and the local feature to obtain an enhanced fused feature and output it to the prediction output layer;

[0026] The prediction output layer is constructed based on the decoder of the RT-DETR model, and is used to map the enhanced fusion features into PCBA solder joint defect detection results carrying defect types and defect locations;

[0027] The formula of the loss function is:

[0028] ;

[0029] in, Represents the loss value of the loss function; represents the defect classification loss; represents the bounding box regression loss; represents the bounding box overlap loss; represents feature consistency loss; 、 、 、 Both represent weight coefficients.

[0030] Furthermore, the step S3 is specifically as follows:

[0031] Dividing the data set into a training set, a validation set, and a test set in a ratio of 8:1:1 using a stratified sampling method, setting hyperparameters including at least a batch size, an optimizer, a learning rate, a weight decay rate, and a training cycle, and training a solder joint defect detection model using the training set and the hyperparameters until a loss value of the loss function is less than a preset loss threshold or a preset early stopping condition is met;

[0032] The average precision mean, Giga floating point operations per second, parameter count, and frames per second are calculated using the validation set to verify the trained solder joint defect detection model. If the verification fails, the training set is expanded to continue training. If the verification passes, then:

[0033] The confidence level is calculated using the test set to test the verified solder joint defect detection model. If the test fails, the training set is expanded to continue training; if the test passes, the training is terminated.

[0034] Furthermore, the step S4 is specifically as follows:

[0035] The solder joint defect detection model that has passed the test is compressed using quantization technology and dynamic pruning technology, and the performance of the compressed solder joint defect detection model is verified. Actual PCBA solder joint images are collected, and drift compensation training is performed on the performance-verified solder joint defect detection model using each of the actual PCBA solder joint images. The solder joint defect detection model after drift compensation training is deployed using containerization technology.

[0036] In a second aspect, the present invention provides a PCBA solder joint defect detection system based on an improved RT-DETR, comprising the following modules:

[0037] A data set construction module is used to obtain a large number of historical PCBA solder joint images, pre-process and annotate each of the historical PCBA solder joint images, and then construct a data set;

[0038] A solder joint defect detection model creation module is used to create a solder joint defect detection model based on the feature extraction layer, the feature fusion layer and the prediction output layer, and set the loss function of the solder joint defect detection model;

[0039] a solder joint defect detection model training module, configured to divide the data set into a training set, a validation set, and a test set, train the solder joint defect detection model using the training set and a loss function, validate the trained solder joint defect detection model using the validation set, and test the validated solder joint defect detection model using the test set;

[0040] A solder joint defect detection model deployment module is used to compress, drift compensate and deploy the solder joint defect detection model that has passed the test;

[0041] The defect detection module is used to perform PCBA solder point defect detection by using the deployed solder point defect detection model.

[0042] Furthermore, the dataset construction module is specifically used to:

[0043] A large number of historical PCBA solder joint images with different lighting conditions, shooting angles, and PCBA models are acquired. Each of the historical PCBA solder joint images is preprocessed, including at least cropping, contrast adjustment, exposure adjustment, and size adjustment. The preprocessed historical PCBA solder joint images are labeled with defect types and defect locations using the labelimg annotation tool. A dataset is constructed based on the labeled historical PCBA solder joint images. The defect types include at least normal, kneeling foot, excessive solder, insufficient solder, and continuous solder.

[0044] Furthermore, in the solder joint defect detection model creation module, the feature extraction layer is constructed based on the backbone network of the RT-DETR model, and is used to extract shallow features and deep features from the input PCBA solder joint image, and input the shallow features and deep features into the feature fusion layer;

[0045] The feature fusion layer is constructed based on the local feature processing module, the global feature processing module and the feature interaction output module;

[0046] The local feature processing module is constructed based on an input channel adjustment unit, a local enhancement unit, a structure optimization unit, and a residual connection unit; the input channel adjustment unit is used to perform channel compression and preliminary feature extraction on the input shallow features and deep features through a 1x1 first convolution normalization activation layer to obtain initial features; the local enhancement unit is used to enhance the local perception ability of the initial features and introduce nonlinear transformation through two stacked local enhancement sub-units to obtain enhanced initial features; the structure optimization unit is used to fuse the two-branch structures of the two local enhancement sub-units into a single 3x3 convolution layer in the inference stage; the residual connection unit is used to fuse the enhanced initial features, shallow features, and deep features to obtain local features, and input the local features into the global feature processing module;

[0047] The global feature processing module is constructed based on an attention generation unit, a channel space mapping unit, and a spatial compression unit; the attention generation unit is used to fuse the local features through global average pooling, global maximum pooling, and a Softmax function to obtain a fused attention feature; the channel space mapping unit is used to perform spatial mapping on the local features through a 1x1 second convolution normalized activation layer to obtain a channel space mapping feature; the spatial compression unit is used to perform spatial compression on the local features through a 1x1 third convolution normalized activation layer to obtain a spatial compression feature;

[0048] The feature interaction output module is used to perform matrix multiplication on the fused attention feature and the channel space mapping feature to obtain a spatial enhancement feature, perform matrix multiplication on the channel space mapping feature and the spatial compression feature to obtain a channel enhancement feature, input the spatial enhancement feature into the 1x1 fourth convolution normalized activation layer to obtain a spatial attention map, and input the channel enhancement feature into the unnormalized convolution activation layer to obtain a channel attention map; perform element-by-element multiplication on the spatial attention map and the channel attention map to obtain an initial fused feature, perform a residual connection on the initial fused feature and the local feature to obtain an enhanced fused feature and output it to the prediction output layer;

[0049] The prediction output layer is constructed based on the decoder of the RT-DETR model, and is used to map the enhanced fusion features into PCBA solder joint defect detection results carrying defect types and defect locations;

[0050] The formula of the loss function is:

[0051] ;

[0052] in, Represents the loss value of the loss function; represents the defect classification loss; represents the bounding box regression loss; represents the bounding box overlap loss; represents feature consistency loss; 、 、 、 Both represent weight coefficients.

[0053] Furthermore, the solder joint defect detection model training module is specifically:

[0054] Dividing the data set into a training set, a validation set, and a test set in a ratio of 8:1:1 using a stratified sampling method, setting hyperparameters including at least a batch size, an optimizer, a learning rate, a weight decay rate, and a training cycle, and training a solder joint defect detection model using the training set and the hyperparameters until a loss value of the loss function is less than a preset loss threshold or a preset early stopping condition is met;

[0055] The average precision mean, Giga floating point operations per second, parameter count, and frames per second are calculated using the validation set to verify the trained solder joint defect detection model. If the verification fails, the training set is expanded to continue training. If the verification passes, then:

[0056] The confidence level is calculated using the test set to test the verified solder joint defect detection model. If the test fails, the training set is expanded to continue training; if the test passes, the training is terminated.

[0057] Furthermore, the solder joint defect detection model deployment module is specifically used to:

[0058] The solder joint defect detection model that has passed the test is compressed using quantization technology and dynamic pruning technology, and the performance of the compressed solder joint defect detection model is verified. Actual PCBA solder joint images are collected, and drift compensation training is performed on the performance-verified solder joint defect detection model using each of the actual PCBA solder joint images. The solder joint defect detection model after drift compensation training is deployed using containerization technology.

[0059] The advantages of the present invention are:

[0060] 1. Build a data set by acquiring a large number of historical PCBA solder joint images; then create a solder joint defect detection model based on the feature extraction layer, feature fusion layer and prediction output layer, and set the loss function of the solder joint defect detection model; divide the data set into training set, validation set and test set, train the solder joint defect detection model through the training set and loss function, verify the trained solder joint defect detection model through the validation set, test the verified solder joint defect detection model through the test set, compress, drift compensate and deploy the tested solder joint defect detection model, and finally perform PCBA solder joint defect detection through the deployed solder joint defect detection model; the feature extraction layer is built based on the backbone network of the RT-DETR model, which is used to extract shallow features and deep features from the input PCBA solder joint image, and input the shallow features and deep features into the feature fusion layer; the feature fusion layer is built based on the local feature processing module, the global feature processing module and the feature interaction output module. It is used to fuse shallow features and deep features to obtain enhanced fusion features and output them to the prediction output layer; the prediction output layer is constructed based on the decoder of the RT-DETR model, which is used to map the enhanced fusion features into PCBA solder joint defect detection results carrying defect types and defect locations; that is, by reconstructing the RepC3 module of the RT-DETR model, replacing the traditional RepC3 module with the feature fusion layer, while retaining the efficient backbone network and decoder of the RT-DETR model to maintain real-time performance, the local feature processing module is used to enhance the perception of small target details, and the attention mechanism of the global feature processing module and the spatial-channel collaborative optimization of the feature interaction output module are combined to effectively bridge the gap between shallow details and deep semantics, and significantly improve the recognition ability of subtle defects in solder joints; at the same time, with the combination of diversified working condition data training, feature consistency loss function constraints and model compression deployment technology, it is finally achieved while meeting the real-time detection requirements and greatly improving the accuracy of PCBA solder joint defect detection.

[0061] 2. The feature fusion layer adopts a unique three-level structure of "local feature processing module + global feature processing module + feature interaction output module", with clear hierarchy and division of labor, which can more effectively integrate feature information of different scales and semantic levels.

[0062] 3. Two stacked local enhancement sub-units enhance the model's perception and nonlinear modeling capabilities for local micro-areas (such as solder joint defects), which is crucial for identifying small solder joint defects (such as kneeling feet and insufficient solder). In the inference stage, the dual-branch structure is fused into a single 3x3 convolutional layer, which significantly improves the inference speed (reduces the amount of computation) while maintaining the enhancement capability during training, which is beneficial for industrial deployment. The enhanced features are residually connected with the original shallow features (containing detailed information) and deep features (containing semantic information), effectively fusing information at different levels of abstraction, so that the local features contain both detailed and semantic information.

[0063] 4. Use global average pooling, maximum pooling and Softmax to generate fused attention features, which can focus on more important areas in the image; through independent channel space mapping and spatial compression operations, feature information of different dimensions can be extracted and refined respectively.

[0064] 5. Perform matrix multiplication on the fused attention features and the channel spatial mapping features to obtain spatial enhancement features, and perform matrix multiplication on the channel spatial mapping features and the spatial compression features to obtain channel enhancement features. This interactive method can better capture the complex relationship between features than simple splicing or addition; further convert the spatial enhancement and channel enhancement features into spatial attention maps and channel attention maps, and perform deep fusion through element-by-element multiplication to form initial fusion features; finally, the initial fusion features after deep interactive fusion are residually connected with the original local features, which not only retains the original information but also incorporates highly refined global perception information. The final output "enhanced fusion features" have a stronger ability to discriminate against minor defects.

[0065] 6. The loss function combines defect classification loss, bounding box regression loss, bounding box overlap loss, and feature consistency loss. The feature consistency loss aims to constrain the feature representation learned by the model to be consistent (possibly for similar normal or defective samples). This helps improve the model's generalization ability, robustness to noise and small changes, and prevent overfitting, thereby improving actual detection accuracy and stability in complex industrial scenarios.

[0066] 7. By collecting images with different lighting conditions, different shooting angles, and different PCBA models, and performing pre-processing such as cropping, contrast / exposure / size adjustment, this greatly enhances the diversity and coverage of the dataset, which is the basis for the model to have strong generalization capabilities; using stratified sampling (8:1:1) to divide the training / validation / test sets helps ensure the representativeness of the data distribution and more objectively evaluate model performance; the verification phase focuses not only on accuracy indicators (such as mean average precision (mAP)), but also on efficiency indicators (thousand frames per second (FPS), giga floating-point operations per second (GFLOPS)) and model size (number of parameters) to ensure that the model meets actual deployment requirements.

[0067] 8. Use quantization and dynamic pruning techniques to compress the trained model, directly optimizing for resource constraints during deployment (such as computing power and memory of embedded devices), reducing computational complexity and storage requirements. Use containerization technology for deployment, which improves deployment convenience, portability, maintainability, and resource utilization, in line with modern industrial software deployment trends. After deployment, the model is trained for drift compensation by collecting actual PCBA solder joint images, effectively coping with changes in the actual production environment that may occur over time (such as equipment aging, material batch differences, subtle environmental changes, etc.), maintaining the model's detection accuracy and stability during long-term operation, and solving the key industrial pain point of model "aging."

[0068] 9. By innovatively constructing a feature fusion layer consisting of a local feature processing module, a global feature processing module, and a feature interaction output module, combined with designs such as stacked local enhancement units, attention mechanisms, and residual connections, the model's local perception ability and feature discrimination accuracy for minor PCBA solder joint defects (such as kneeling feet and insufficient solder) are significantly improved; the introduction of a four-element loss function containing feature consistency loss and a diversified dataset construction strategy enhances the robustness and generalization of the model in complex industrial scenarios; at the same time, by optimizing the structure in the inference stage (such as fusing two branches into a single convolution), adopting quantized pruning and compression technology, containerized deployment, and drift compensation training, it effectively balances detection accuracy and real-time efficiency, and ensures the stability and adaptability of the model in long-term industrial applications, making it particularly suitable for high-precision and high-efficiency PCBA solder joint defect detection needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0070] Figure 1 The present invention is a flow chart of a PCBA solder joint defect detection method based on improved RT-DETR.

[0071] Figure 2 It is a structural schematic diagram of a PCBA solder joint defect detection system based on improved RT-DETR of the present invention.

[0072] Figure 3 It is a structural schematic diagram of the feature fusion layer of the present invention.

[0073] Figure 4 It is a structural schematic diagram of the structural optimization unit of the present invention.

[0074] Figure 5 It is a structural diagram of the traditional RT-DETR model.

[0075] Figure 6This is a comparative thermal diagram of the solder joint defect detection model of the present invention and the traditional RT-DETR model. DETAILED DESCRIPTION

[0076] The technical solution in the embodiments of the present application has the following overall idea: reconstructing the RepC3 module of the RT-DETR model, replacing the traditional RepC3 module with a feature fusion layer, and using a local feature processing module to enhance the perception of small target details while retaining the efficient backbone network and decoder of the RT-DETR model to maintain real-time performance. Combining the attention mechanism of the global feature processing module with the spatial-channel collaborative optimization of the feature interaction output module effectively bridges the gap between shallow details and deep semantics, and significantly improves the ability to identify subtle defects in solder joints. At the same time, combined with diversified working condition data training, feature consistency loss function constraints and model compression deployment technology, it is possible to achieve the improvement of PCBA solder joint defect detection accuracy while meeting real-time detection requirements.

[0077] Please refer to Figures 1 to 6 As shown, a preferred embodiment of the PCBA solder joint defect detection method based on improved RT-DETR of the present invention includes the following steps:

[0078] Step S1: Acquire a large number of historical PCBA solder joint images, pre-process and annotate each of the historical PCBA solder joint images, and then construct a data set;

[0079] Step S2: creating a solder joint defect detection model based on the feature extraction layer, the feature fusion layer, and the prediction output layer, and setting a loss function of the solder joint defect detection model;

[0080] The feature extraction layer is constructed based on the backbone network of the RT-DETR model, and is used to extract shallow features and deep features from the input PCBA solder joint image, and input the shallow features and deep features into the feature fusion layer;

[0081] The feature fusion layer is constructed based on the local feature processing module, the global feature processing module and the feature interaction output module;

[0082] The feature fusion layer adopts a unique three-level structure of "local feature processing module + global feature processing module + feature interaction output module", with clear hierarchy and division of labor, which can more effectively integrate feature information of different scales and semantic levels.

[0083] The local feature processing module is constructed based on an input channel adjustment unit, a local enhancement unit, a structure optimization unit, and a residual connection unit; the input channel adjustment unit is used to perform channel compression and preliminary feature extraction on the input shallow features and deep features through a 1x1 first convolution normalization activation layer (1×1CBS, consisting of a convolution operation, batch normalization, and SiLU activation function) to obtain initial features; the local enhancement unit is used to perform local perception on the initial features through two stacked local enhancement sub-units (RepLEBlock, comprising two parallel 3×3CBS branches and a 1×1CBS branch, the outputs of which are processed by a nonlinear activation function (SiLU) after element-by-element addition. Enhance and introduce nonlinear transformation to obtain enhanced initial features; the structural optimization unit is used to fuse the two-branch structures of the two local enhancement sub-units into a single 3x3 convolution layer during the inference phase (by folding the convolution kernel and the batch normalization parameter (i.e., fusing the convolution and normalization operations), and zero-padding the 1×1 convolution kernel to expand it to a 3×3 size, so that the two branches can be merged into a standard convolution operation); the residual connection unit is used to fuse the enhanced initial features, shallow features, and deep features to obtain local features (a four-dimensional feature tensor of size [n, c, h, w], where n represents the batch size, c represents the channel, and h and w represent the height and width of the feature map, respectively), and input the local features into the global feature processing module;

[0084] Two stacked local enhancement sub-units enhance the model's perception and nonlinear modeling capabilities for local micro-areas (such as solder joint defects), which is crucial for identifying small solder joint defects (such as kneeling feet and insufficient solder). In the inference stage, the dual-branch structure is fused into a single 3x3 convolutional layer, which significantly improves the inference speed (reduces the amount of computation) while maintaining the enhancement capabilities during training, which is beneficial for industrial deployment. The enhanced features are residually connected with the original shallow features (containing detailed information) and deep features (containing semantic information), effectively fusing information at different levels of abstraction so that local features contain both detailed and semantic information.

[0085] The global feature processing module is constructed based on the attention generation unit, the channel space mapping unit and the space compression unit; the attention generation unit is used to fuse the local features through global average pooling, global maximum pooling and Softmax function to obtain fused attention features (the input features (local features) are respectively sent to the two sub-branches, and global average pooling and global maximum pooling operations are performed on the two sub-branches to generate two channel compression features with a size of [n, c, 1, 1], and then the Softmax function is applied to the two features respectively, and the channel dimension (i.e., the second dimension) is rearranged to the last dimension, that is, the size is adjusted to [n, 1, 1, c], and then the two features are spliced ​​on the channel dimension (i.e., the second dimension) to obtain a fused attention feature with a size of [n, 2, 1, c]). The channel space mapping unit is used to perform spatial mapping on each of the local features through a 1x1 second convolution normalization activation layer to obtain a channel space mapping feature (that is, the input feature first passes through a 1×1 CBS to obtain a tensor [n, c, h, w] with the same shape as the original input, and then expands the tensor into [n, c, 1, hw] in the spatial dimension and rearranges it into [n, 1, c, hw], where hw = h×w); the spatial compression unit is used to perform spatial compression on each of the local features through a 1x1 third convolution normalization activation layer to obtain a spatial compression feature (that is, the input feature first passes through a 1×1 CBS to obtain a feature map with an output dimension of [n, 1, h, w], and then flattens its spatial dimension to [n, 1, 1, hw] and adjusts it to [n, 1, hw, 1]);

[0086] Global average pooling, maximum pooling and Softmax are used to generate fused attention features, which can focus on more important areas in the image; through independent channel space mapping and spatial compression operations, feature information of different dimensions can be extracted and refined respectively.

[0087] The feature interaction output module is used to perform matrix multiplication on the fusion attention feature and the channel space mapping feature to obtain a spatial enhancement feature (matrix multiplication of [n,2,1,c] and [n,1,c,hw] is performed to obtain a tensor of size [n,2,1,hw], and the shape is adjusted to [n,2,h,w]), and to perform matrix multiplication on the channel space mapping feature and the spatial compression feature to obtain a channel enhancement feature (matrix multiplication of [n,1,c,hw] and [n,1,hw,1] is performed to obtain a tensor of size [n,1,c,1], and the shape is adjusted to [n,c,1,1]), and the spatial enhancement feature is input into the fourth convolution normalized activation layer of 1x1 to obtain a spatial attention map (the spatial enhancement feature [ n,2,h,w] is fed into a 1×1 CBS to obtain a spatial attention map of dimension [n,1,h,w]), the channel enhancement feature is input into an unnormalized convolution activation layer (composed of a convolution operation and a SiLU activation function) to obtain a channel attention map (the channel enhancement feature [n,c,1,1] is fed into an unnormalized convolution activation layer (CS) to obtain an enhanced channel attention map of dimension [n,c,1,1]); the spatial attention map and the channel attention map are element-wise multiplied to obtain an initial fusion feature (shape [n,c,h,w]), the initial fusion feature is residually connected with the local feature to obtain an enhanced fusion feature (shape still [n,c,h,w]) and output to the prediction output layer;

[0088] Matrix multiplication is performed on the fused attention features and the channel space mapping features to obtain spatial enhancement features, and matrix multiplication is performed on the channel space mapping features and the spatial compression features to obtain channel enhancement features. This interactive method can better capture the complex relationship between features than simple splicing or addition; the spatial enhancement and channel enhancement features are further converted into spatial attention maps and channel attention maps, and deep fusion is performed through element-by-element multiplication to form initial fusion features; finally, the initial fusion features after deep interactive fusion are residually connected with the original local features, which not only retains the original information but also incorporates highly refined global perception information. The final output "enhanced fusion features" have a stronger ability to discriminate against minor defects.

[0089] The prediction output layer is constructed based on the decoder of the RT-DETR model, and is used to map the enhanced fusion features into PCBA solder joint defect detection results carrying defect types and defect locations;

[0090] Step S3, dividing the data set into a training set, a validation set, and a test set, training a solder joint defect detection model using the training set and a loss function, validating the trained solder joint defect detection model using the validation set, and testing the validated solder joint defect detection model using the test set;

[0091] Step S4: compressing, drift compensation training, and deploying the solder joint defect detection model that has passed the test;

[0092] Step S5: Perform PCBA solder point defect detection using the deployed solder point defect detection model.

[0093] The step S1 is specifically as follows:

[0094] A large number of historical PCBA solder joint images with different lighting conditions, shooting angles, and PCBA models are acquired. Each of the historical PCBA solder joint images is preprocessed, including at least cropping, contrast adjustment, exposure adjustment, and size adjustment. The preprocessed historical PCBA solder joint images are labeled with defect types and defect locations using the labelimg annotation tool. A dataset is constructed based on the labeled historical PCBA solder joint images. The defect types include at least normal, kneeling foot, excessive solder, insufficient solder, and continuous solder.

[0095] In step S2, the formula of the loss function is:

[0096] ;

[0097] in, Represents the loss value of the loss function; represents the defect classification loss; represents the bounding box regression loss; represents the bounding box overlap loss; represents feature consistency loss; 、 、 、 Both represent weight coefficients.

[0098] ;

[0099] in, represents the number of positive samples (the number of predictions that match the true bounding box); represents the predicted defect category probability vector; Represents the true category one-hot vector; represents the focal loss function;

[0100] ;

[0101] in, Represents the predicted normalized bounding box coordinates; represents the real bounding box coordinates; represents the L1 norm;

[0102] ;

[0103] ;

[0104] in, represents the spatial attention map output; Represents the channel attention map output; Represents the input feature tensor (H×W×C); represents the generalized intersection-over-union loss function.

[0105] The loss function combines defect classification loss, bounding box regression loss, bounding box overlap loss, and feature consistency loss. The feature consistency loss aims to constrain the feature representation learned by the model to be consistent (possibly for similar normal or defective samples). This helps improve the model's generalization ability, robustness to noise and small changes, and prevent overfitting, thereby improving actual detection accuracy and stability in complex industrial scenarios.

[0106] The step S3 is specifically as follows:

[0107] The data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1 by a stratified sampling method. Hyperparameters including at least the number of batch training, the optimizer, the learning rate, the weight decay rate, and the training cycle are set. The solder joint defect detection model is trained using the training set and the hyperparameters until the loss value of the loss function is less than a preset loss threshold or the preset early stopping condition is met. The number of batch training is 8; the optimizer uses AdamW; the learning rate is 0.0001; the weight decay rate is 0.0001; and the training cycle is 216 rounds.

[0108] The mean average precision (mAP), gigaflops (GFLOPs), parameters (Params), and frames per second (FPS) are calculated using the validation set to verify the trained solder joint defect detection model. If the verification fails, the training set is expanded to continue training. If the verification passes, then:

[0109] The confidence level is calculated using the test set to test the verified solder joint defect detection model. If the test fails, the training set is expanded to continue training; if the test passes, the training is terminated.

[0110] The step S4 is specifically as follows:

[0111] The solder joint defect detection model that has passed the test is compressed using quantization technology and dynamic pruning technology, and the performance of the compressed solder joint defect detection model is verified. Actual PCBA solder joint images are collected, and drift compensation training is performed on the performance-verified solder joint defect detection model using each of the actual PCBA solder joint images. The solder joint defect detection model after drift compensation training is deployed using containerization technology.

[0112] Quantization and dynamic pruning techniques are used to compress the trained model, directly optimizing for resource constraints during deployment (such as computing power and memory of embedded devices), reducing computational complexity and storage requirements. Containerization technology is used for deployment, improving deployment convenience, portability, maintainability, and resource utilization, in line with modern industrial software deployment trends. After deployment, the model is trained for drift compensation by collecting actual PCBA solder joint images, effectively responding to changes in the actual production environment that may occur over time (such as equipment aging, material batch differences, subtle environmental changes, etc.), maintaining the model's detection accuracy and stability during long-term operation, and resolving the key industrial pain point of model "aging."

[0113] By innovatively constructing a feature fusion layer consisting of a local feature processing module, a global feature processing module, and a feature interaction output module, combined with designs such as stacked local enhancement units, an attention mechanism, and residual connections, the model's local perception ability and feature discrimination accuracy for minor PCBA solder joint defects (such as kneeling feet and insufficient solder) are significantly improved; the introduction of a four-element loss function including feature consistency loss and a diversified dataset construction strategy enhances the robustness and generalization of the model in complex industrial scenarios; at the same time, by optimizing the structure in the inference stage (such as fusing two branches into a single convolution), adopting quantized pruning and compression technology, containerized deployment, and drift compensation training, it effectively balances detection accuracy and real-time efficiency, and ensures the stability and adaptability of the model in long-term industrial applications. It is particularly suitable for high-precision and high-efficiency PCBA solder joint defect detection needs.

[0114] A preferred embodiment of the PCBA solder joint defect detection system based on the improved RT-DETR of the present invention includes the following modules:

[0115] A data set construction module is used to obtain a large number of historical PCBA solder joint images, pre-process and annotate each of the historical PCBA solder joint images, and then construct a data set;

[0116] A solder joint defect detection model creation module is used to create a solder joint defect detection model based on the feature extraction layer, the feature fusion layer and the prediction output layer, and set the loss function of the solder joint defect detection model;

[0117] The feature extraction layer is constructed based on the backbone network of the RT-DETR model, and is used to extract shallow features and deep features from the input PCBA solder joint image, and input the shallow features and deep features into the feature fusion layer;

[0118] The feature fusion layer is constructed based on the local feature processing module, the global feature processing module and the feature interaction output module;

[0119] The feature fusion layer adopts a unique three-level structure of "local feature processing module + global feature processing module + feature interaction output module", with clear hierarchy and division of labor, which can more effectively integrate feature information of different scales and semantic levels.

[0120] The local feature processing module is constructed based on an input channel adjustment unit, a local enhancement unit, a structure optimization unit, and a residual connection unit; the input channel adjustment unit is used to perform channel compression and preliminary feature extraction on the input shallow features and deep features through a 1x1 first convolution normalization activation layer (1×1CBS, consisting of a convolution operation, batch normalization, and SiLU activation function) to obtain initial features; the local enhancement unit is used to perform local perception on the initial features through two stacked local enhancement sub-units (RepLEBlock, comprising two parallel 3×3CBS branches and a 1×1CBS branch, the outputs of which are processed by a nonlinear activation function (SiLU) after element-by-element addition. Enhance and introduce nonlinear transformation to obtain enhanced initial features; the structural optimization unit is used to fuse the two-branch structures of the two local enhancement sub-units into a single 3x3 convolution layer during the inference phase (by folding the convolution kernel and the batch normalization parameter (i.e., fusing the convolution and normalization operations), and zero-padding the 1×1 convolution kernel to expand it to a 3×3 size, so that the two branches can be merged into a standard convolution operation); the residual connection unit is used to fuse the enhanced initial features, shallow features, and deep features to obtain local features (a four-dimensional feature tensor of size [n, c, h, w], where n represents the batch size, c represents the channel, and h and w represent the height and width of the feature map, respectively), and input the local features into the global feature processing module;

[0121] Two stacked local enhancement sub-units enhance the model's perception and nonlinear modeling capabilities for local micro-areas (such as solder joint defects), which is crucial for identifying small solder joint defects (such as kneeling feet and insufficient solder). In the inference stage, the dual-branch structure is fused into a single 3x3 convolutional layer, which significantly improves the inference speed (reduces the amount of computation) while maintaining the enhancement capabilities during training, which is beneficial for industrial deployment. The enhanced features are residually connected with the original shallow features (containing detailed information) and deep features (containing semantic information), effectively fusing information at different levels of abstraction so that local features contain both detailed and semantic information.

[0122] The global feature processing module is constructed based on the attention generation unit, the channel space mapping unit and the space compression unit; the attention generation unit is used to fuse the local features through global average pooling, global maximum pooling and Softmax function to obtain fused attention features (the input features (local features) are respectively sent to the two sub-branches, and global average pooling and global maximum pooling operations are performed on the two sub-branches to generate two channel compression features with a size of [n, c, 1, 1], and then the Softmax function is applied to the two features respectively, and the channel dimension (i.e., the second dimension) is rearranged to the last dimension, that is, the size is adjusted to [n, 1, 1, c], and then the two features are spliced ​​on the channel dimension (i.e., the second dimension) to obtain a fused attention feature with a size of [n, 2, 1, c]). The channel space mapping unit is used to perform spatial mapping on each of the local features through a 1x1 second convolution normalization activation layer to obtain a channel space mapping feature (that is, the input feature first passes through a 1×1 CBS to obtain a tensor [n, c, h, w] with the same shape as the original input, and then expands the tensor into [n, c, 1, hw] in the spatial dimension and rearranges it into [n, 1, c, hw], where hw = h×w); the spatial compression unit is used to perform spatial compression on each of the local features through a 1x1 third convolution normalization activation layer to obtain a spatial compression feature (that is, the input feature first passes through a 1×1 CBS to obtain a feature map with an output dimension of [n, 1, h, w], and then flattens its spatial dimension to [n, 1, 1, hw] and adjusts it to [n, 1, hw, 1]);

[0123] Global average pooling, maximum pooling and Softmax are used to generate fused attention features, which can focus on more important areas in the image; through independent channel space mapping and spatial compression operations, feature information of different dimensions can be extracted and refined respectively.

[0124] The feature interaction output module is used to perform matrix multiplication on the fusion attention feature and the channel space mapping feature to obtain a spatial enhancement feature (matrix multiplication of [n,2,1,c] and [n,1,c,hw] is performed to obtain a tensor of size [n,2,1,hw], and the shape is adjusted to [n,2,h,w]), and to perform matrix multiplication on the channel space mapping feature and the spatial compression feature to obtain a channel enhancement feature (matrix multiplication of [n,1,c,hw] and [n,1,hw,1] is performed to obtain a tensor of size [n,1,c,1], and the shape is adjusted to [n,c,1,1]), and the spatial enhancement feature is input into the fourth convolution normalized activation layer of 1x1 to obtain a spatial attention map (the spatial enhancement feature [ n,2,h,w] is fed into a 1×1 CBS to obtain a spatial attention map of dimension [n,1,h,w]), the channel enhancement feature is input into an unnormalized convolution activation layer (composed of a convolution operation and a SiLU activation function) to obtain a channel attention map (the channel enhancement feature [n,c,1,1] is fed into an unnormalized convolution activation layer (CS) to obtain an enhanced channel attention map of dimension [n,c,1,1]); the spatial attention map and the channel attention map are element-wise multiplied to obtain an initial fusion feature (shape [n,c,h,w]), the initial fusion feature is residually connected with the local feature to obtain an enhanced fusion feature (shape still [n,c,h,w]) and output to the prediction output layer;

[0125] Matrix multiplication is performed on the fused attention features and the channel space mapping features to obtain spatial enhancement features, and matrix multiplication is performed on the channel space mapping features and the spatial compression features to obtain channel enhancement features. This interactive method can better capture the complex relationship between features than simple splicing or addition; the spatial enhancement and channel enhancement features are further converted into spatial attention maps and channel attention maps, and deep fusion is performed through element-by-element multiplication to form initial fusion features; finally, the initial fusion features after deep interactive fusion are residually connected with the original local features, which not only retains the original information but also incorporates highly refined global perception information. The final output "enhanced fusion features" have a stronger ability to discriminate against minor defects.

[0126] The prediction output layer is constructed based on the decoder of the RT-DETR model, and is used to map the enhanced fusion features into PCBA solder joint defect detection results carrying defect types and defect locations;

[0127] a solder joint defect detection model training module, configured to divide the data set into a training set, a validation set, and a test set, train the solder joint defect detection model using the training set and a loss function, validate the trained solder joint defect detection model using the validation set, and test the validated solder joint defect detection model using the test set;

[0128] A solder joint defect detection model deployment module is used to compress, drift compensate and deploy the solder joint defect detection model that has passed the test;

[0129] The defect detection module is used to perform PCBA solder point defect detection by using the deployed solder point defect detection model.

[0130] The dataset construction module is specifically used for:

[0131] A large number of historical PCBA solder joint images with different lighting conditions, shooting angles, and PCBA models are acquired. Each of the historical PCBA solder joint images is preprocessed, including at least cropping, contrast adjustment, exposure adjustment, and size adjustment. The preprocessed historical PCBA solder joint images are labeled with defect types and defect locations using the labelimg annotation tool. A dataset is constructed based on the labeled historical PCBA solder joint images. The defect types include at least normal, kneeling foot, excessive solder, insufficient solder, and continuous solder.

[0132] In the solder joint defect detection model creation module, the formula of the loss function is:

[0133] ;

[0134] in, Represents the loss value of the loss function; represents the defect classification loss; represents the bounding box regression loss; represents the bounding box overlap loss; represents feature consistency loss; 、 、 、 Both represent weight coefficients.

[0135] ;

[0136] in, represents the number of positive samples (the number of predictions that match the true bounding box); represents the predicted defect category probability vector; Represents the true category one-hot vector; represents the focal loss function;

[0137] ;

[0138] in, Represents the predicted normalized bounding box coordinates; represents the real bounding box coordinates; represents the L1 norm;

[0139] ;

[0140] ;

[0141] in, represents the spatial attention map output; Represents the channel attention map output; Represents the input feature tensor (H×W×C); represents the generalized intersection-over-union loss function.

[0142] The loss function combines defect classification loss, bounding box regression loss, bounding box overlap loss, and feature consistency loss. The feature consistency loss aims to constrain the feature representation learned by the model to be consistent (possibly for similar normal or defective samples). This helps improve the model's generalization ability, robustness to noise and small changes, and prevent overfitting, thereby improving actual detection accuracy and stability in complex industrial scenarios.

[0143] The solder joint defect detection model training module is specifically:

[0144] The data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1 by a stratified sampling method. Hyperparameters including at least the number of batch training, the optimizer, the learning rate, the weight decay rate, and the training cycle are set. The solder joint defect detection model is trained using the training set and the hyperparameters until the loss value of the loss function is less than a preset loss threshold or the preset early stopping condition is met. The number of batch training is 8; the optimizer uses AdamW; the learning rate is 0.0001; the weight decay rate is 0.0001; and the training cycle is 216 rounds.

[0145] The mean average precision (mAP), gigaflops (GFLOPs), parameters (Params), and frames per second (FPS) are calculated using the validation set to verify the trained solder joint defect detection model. If the verification fails, the training set is expanded to continue training. If the verification passes, then:

[0146] The confidence level is calculated using the test set to test the verified solder joint defect detection model. If the test fails, the training set is expanded to continue training; if the test passes, the training is terminated.

[0147] The solder joint defect detection model deployment module is specifically used to:

[0148] The solder joint defect detection model that has passed the test is compressed using quantization technology and dynamic pruning technology, and the performance of the compressed solder joint defect detection model is verified. Actual PCBA solder joint images are collected, and drift compensation training is performed on the performance-verified solder joint defect detection model using each of the actual PCBA solder joint images. The solder joint defect detection model after drift compensation training is deployed using containerization technology.

[0149] Quantization and dynamic pruning techniques are used to compress the trained model, directly optimizing for resource constraints during deployment (such as computing power and memory of embedded devices), reducing computational complexity and storage requirements. Containerization technology is used for deployment, improving deployment convenience, portability, maintainability, and resource utilization, in line with modern industrial software deployment trends. After deployment, the model is trained for drift compensation by collecting actual PCBA solder joint images, effectively responding to changes in the actual production environment that may occur over time (such as equipment aging, material batch differences, subtle environmental changes, etc.), maintaining the model's detection accuracy and stability during long-term operation, and resolving the key industrial pain point of model "aging."

[0150] By innovatively constructing a feature fusion layer consisting of a local feature processing module, a global feature processing module, and a feature interaction output module, combined with designs such as stacked local enhancement units, an attention mechanism, and residual connections, the model's local perception ability and feature discrimination accuracy for minor PCBA solder joint defects (such as kneeling feet and insufficient solder) are significantly improved; the introduction of a four-element loss function including feature consistency loss and a diversified dataset construction strategy enhances the robustness and generalization of the model in complex industrial scenarios; at the same time, by optimizing the structure in the inference stage (such as fusing two branches into a single convolution), adopting quantized pruning and compression technology, containerized deployment, and drift compensation training, it effectively balances detection accuracy and real-time efficiency, and ensures the stability and adaptability of the model in long-term industrial applications. It is particularly suitable for high-precision and high-efficiency PCBA solder joint defect detection needs.

[0151] The present invention is compared with the prior art detection methods of Faster-RCNN, SSD, Deformable DETR, and RT-DETR-R50. The comparison results are as follows:

[0152]

[0153] Among them, mAP 50 Meaning: Select IoU=0.5, the average precision detection indicator of the validation set data; mAP 75 Meaning: Select IoU=0.75, the average precision detection index of the validation set data; mAP 50:95Meaning: Select 0.5 < IoU < 0.95, the average precision detection index of the validation set data; Meaning of Params (MB): The size of the detection model parameters, in MB; GFLOPs meaning: The number of gigafloating-point operations per second performed by the detection model; FPS (f / s) meaning: The number of frames that can be detected per second.

[0154] From the comparison results, it can be seen that the present invention shows significant performance advantages in the PCBA multi-type solder joint defect detection task. In terms of accuracy, the present invention reaches 79.6%, 79.2% and 63.8% respectively in the three indexes of mAP 50 、mAP 75 and mAP 50:95 , all of which are better than the original RT-DETR-R50 and other comparison models. Especially, the optimal performance is achieved in mAP 75 , indicating stronger precise positioning ability in complex solder joint areas. Compared with RT-DETR-R50, while ensuring the improvement of accuracy, the parameter quantity of the model of the present invention is reduced from 49.80 MB to 45.65 MB, and GFLOPs is reduced from 63.92 to 58.70, indicating higher model efficiency and lower computing resource consumption; at the same time, the inference speed is also slightly improved, reaching 74.09 FPS, further verifying the deployment feasibility and practical value of the present invention in actual industrial detection applications.

[0155] In order to further verify the improvement effect of the present invention on the ability to focus on solder joint targets, heatmap visualization analysis was carried out on the feature responses on the test images. When the original RT-DETR was not improved, the response areas in the heatmap were relatively scattered, and some solder joint defects were not effectively activated, reflecting insufficient attention ability in the small target solder joint defect detection task. The overall response of the present invention in the solder joint area is significantly enhanced, and all solder joint areas are responded, indicating good performance in integrating local details and global semantic features, which helps to improve the expression quality of multi-scale features.

[0156] To sum up, the advantages of the present invention are as follows:

[0157] 1. Build a data set by acquiring a large number of historical PCBA solder joint images; then create a solder joint defect detection model based on the feature extraction layer, feature fusion layer and prediction output layer, and set the loss function of the solder joint defect detection model; divide the data set into training set, validation set and test set, train the solder joint defect detection model through the training set and loss function, verify the trained solder joint defect detection model through the validation set, test the verified solder joint defect detection model through the test set, compress, drift compensate and deploy the tested solder joint defect detection model, and finally perform PCBA solder joint defect detection through the deployed solder joint defect detection model; the feature extraction layer is built based on the backbone network of the RT-DETR model, which is used to extract shallow features and deep features from the input PCBA solder joint image, and input the shallow features and deep features into the feature fusion layer; the feature fusion layer is built based on the local feature processing module, the global feature processing module and the feature interaction output module. It is used to fuse shallow features and deep features to obtain enhanced fusion features and output them to the prediction output layer; the prediction output layer is constructed based on the decoder of the RT-DETR model, which is used to map the enhanced fusion features into PCBA solder joint defect detection results carrying defect types and defect locations; that is, by reconstructing the RepC3 module of the RT-DETR model, replacing the traditional RepC3 module with the feature fusion layer, while retaining the efficient backbone network and decoder of the RT-DETR model to maintain real-time performance, the local feature processing module is used to enhance the perception of small target details, and the attention mechanism of the global feature processing module and the spatial-channel collaborative optimization of the feature interaction output module are combined to effectively bridge the gap between shallow details and deep semantics, and significantly improve the recognition ability of subtle defects in solder joints; at the same time, with the combination of diversified working condition data training, feature consistency loss function constraints and model compression deployment technology, it is finally achieved while meeting the real-time detection requirements and greatly improving the accuracy of PCBA solder joint defect detection.

[0158] 2. The feature fusion layer adopts a unique three-level structure of "local feature processing module + global feature processing module + feature interaction output module", with clear hierarchy and division of labor, which can more effectively integrate feature information of different scales and semantic levels.

[0159] 3. Two stacked local enhancement sub-units enhance the model's perception and nonlinear modeling capabilities for local micro-areas (such as solder joint defects), which is crucial for identifying small solder joint defects (such as kneeling feet and insufficient solder). In the inference stage, the dual-branch structure is fused into a single 3x3 convolutional layer, which significantly improves the inference speed (reduces the amount of computation) while maintaining the enhancement capability during training, which is beneficial for industrial deployment. The enhanced features are residually connected with the original shallow features (containing detailed information) and deep features (containing semantic information), effectively fusing information at different levels of abstraction, so that the local features contain both detailed and semantic information.

[0160] 4. Use global average pooling, maximum pooling and Softmax to generate fused attention features, which can focus on more important areas in the image; through independent channel space mapping and spatial compression operations, feature information of different dimensions can be extracted and refined respectively.

[0161] 5. Perform matrix multiplication on the fused attention features and the channel spatial mapping features to obtain spatial enhancement features, and perform matrix multiplication on the channel spatial mapping features and the spatial compression features to obtain channel enhancement features. This interactive method can better capture the complex relationship between features than simple splicing or addition; further convert the spatial enhancement and channel enhancement features into spatial attention maps and channel attention maps, and perform deep fusion through element-by-element multiplication to form initial fusion features; finally, the initial fusion features after deep interactive fusion are residually connected with the original local features, which not only retains the original information but also incorporates highly refined global perception information. The final output "enhanced fusion features" have a stronger ability to discriminate against minor defects.

[0162] 6. The loss function combines defect classification loss, bounding box regression loss, bounding box overlap loss, and feature consistency loss. The feature consistency loss aims to constrain the feature representation learned by the model to be consistent (possibly for similar normal or defective samples). This helps improve the model's generalization ability, robustness to noise and small changes, and prevent overfitting, thereby improving actual detection accuracy and stability in complex industrial scenarios.

[0163] 7. By collecting images with different lighting conditions, different shooting angles, and different PCBA models, and performing pre-processing such as cropping, contrast / exposure / size adjustment, this greatly enhances the diversity and coverage of the dataset, which is the basis for the model to have strong generalization capabilities; using stratified sampling (8:1:1) to divide the training / validation / test sets helps ensure the representativeness of the data distribution and more objectively evaluate model performance; the verification phase focuses not only on accuracy indicators (such as mean average precision (mAP)), but also on efficiency indicators (thousand frames per second (FPS), giga floating-point operations per second (GFLOPS)) and model size (number of parameters) to ensure that the model meets actual deployment requirements.

[0164] 8. Use quantization and dynamic pruning techniques to compress the trained model, directly optimizing for resource constraints during deployment (such as computing power and memory of embedded devices), reducing computational complexity and storage requirements. Use containerization technology for deployment, which improves deployment convenience, portability, maintainability, and resource utilization, in line with modern industrial software deployment trends. After deployment, the model is trained for drift compensation by collecting actual PCBA solder joint images, effectively coping with changes in the actual production environment that may occur over time (such as equipment aging, material batch differences, subtle environmental changes, etc.), maintaining the model's detection accuracy and stability during long-term operation, and solving the key industrial pain point of model "aging."

[0165] 9. By innovatively constructing a feature fusion layer consisting of a local feature processing module, a global feature processing module, and a feature interaction output module, combined with designs such as stacked local enhancement units, attention mechanisms, and residual connections, the model's local perception ability and feature discrimination accuracy for minor PCBA solder joint defects (such as kneeling feet and insufficient solder) are significantly improved; the introduction of a four-element loss function containing feature consistency loss and a diversified dataset construction strategy enhances the robustness and generalization of the model in complex industrial scenarios; at the same time, by optimizing the structure in the inference stage (such as fusing two branches into a single convolution), adopting quantized pruning and compression technology, containerized deployment, and drift compensation training, it effectively balances detection accuracy and real-time efficiency, and ensures the stability and adaptability of the model in long-term industrial applications, making it particularly suitable for high-precision and high-efficiency PCBA solder joint defect detection needs.

[0166] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A PCBA solder joint defect detection method based on improved RT-DETR, characterized by: The steps include: Step S1: Acquire a large number of historical PCBA solder joint images, pre-process and annotate each of the historical PCBA solder joint images, and then construct a data set; Step S2: creating a solder joint defect detection model based on the feature extraction layer, the feature fusion layer, and the prediction output layer, and setting a loss function of the solder joint defect detection model; The feature extraction layer is built based on the backbone network of the RT-DETR model. It is used to extract shallow features and deep features from the input PCBA solder joint image and input the shallow features and deep features into the feature fusion layer. The feature fusion layer is constructed based on the local feature processing module, the global feature processing module and the feature interaction output module; The local feature processing module is constructed based on the input channel adjustment unit, the local enhancement unit, the structure optimization unit, and the residual connection unit. The input channel adjustment unit is used to perform channel compression and preliminary feature extraction on the input shallow and deep features through the first 1x1 convolution normalization activation layer to obtain the initial features. The local enhancement unit is used to enhance the local perception ability of the initial features and introduce nonlinear transformation through two stacked local enhancement sub-units to obtain enhanced initial features. The structure optimization unit is used to fuse the two-branch structure of the two local enhancement sub-units into a single 3x3 convolutional layer during the inference phase. The residual connection unit is used to fuse the enhanced initial features, shallow features, and deep features to obtain local features, which are then input into the global feature processing module. The global feature processing module is constructed based on the attention generation unit, the channel space mapping unit and the spatial compression unit; the attention generation unit is used to fuse the local features through global average pooling, global maximum pooling and Softmax function to obtain the fused attention feature; the channel space mapping unit is used to spatially map the local features through the second 1x1 convolution normalization activation layer to obtain the channel space mapping feature; the spatial compression unit is used to spatially compress the local features through the third 1x1 convolution normalization activation layer to obtain the spatial compression feature; The feature interaction output module is used to perform matrix multiplication on the fusion attention feature and the channel space mapping feature to obtain the spatial enhancement feature, and to perform matrix multiplication on the channel space mapping feature and the spatial compression feature to obtain the channel enhancement feature. The spatial enhancement feature is input into the 1x1 fourth convolution normalized activation layer to obtain the spatial attention map, and the channel enhancement feature is input into the unnormalized convolution activation layer to obtain the channel attention map. The spatial attention map and the channel attention map are element-wise multiplied to obtain the initial fusion feature, and the initial fusion feature is residually connected with the local feature to obtain the enhanced fusion feature and output it to the prediction output layer; Step S3, dividing the data set into a training set, a validation set, and a test set, training a solder joint defect detection model using the training set and a loss function, validating the trained solder joint defect detection model using the validation set, and testing the validated solder joint defect detection model using the test set; Step S4: compressing, drift compensation training, and deploying the solder joint defect detection model that has passed the test; Step S5: Perform PCBA solder point defect detection using the deployed solder point defect detection model.

2. A PCBA solder joint defect detection method based on improved RT-DETR according to claim 1, characterized in that: The step S1 is specifically as follows: A large number of historical PCBA solder joint images with different lighting conditions, shooting angles, and PCBA models are acquired. Each of the historical PCBA solder joint images is preprocessed, including at least cropping, contrast adjustment, exposure adjustment, and size adjustment. The preprocessed historical PCBA solder joint images are labeled with defect types and defect locations using the labelimg annotation tool. A dataset is constructed based on the labeled historical PCBA solder joint images. The defect types include at least normal, kneeling foot, excessive solder, insufficient solder, and continuous solder.

3. A PCBA solder joint defect detection method based on improved RT-DETR as claimed in claim 1, characterized in that: The prediction output layer is constructed based on the decoder of the RT-DETR model, and is used to map the enhanced fusion features into PCBA solder joint defect detection results carrying defect types and defect locations; The formula of the loss function is: ； in, Represents the loss value of the loss function; represents the defect classification loss; represents the bounding box regression loss; represents the bounding box overlap loss; represents feature consistency loss; 、 、 、 Both represent weight coefficients.

4. A PCBA solder joint defect detection method based on improved RT-DETR as claimed in claim 1, characterized in that: The step S3 is specifically as follows: Dividing the data set into a training set, a validation set, and a test set in a ratio of 8:1:1 using a stratified sampling method, setting hyperparameters including at least a batch size, an optimizer, a learning rate, a weight decay rate, and a training cycle, and training a solder joint defect detection model using the training set and the hyperparameters until a loss value of the loss function is less than a preset loss threshold or a preset early stopping condition is met; The average precision mean, Giga floating point operations per second, parameter count, and frames per second are calculated using the validation set to verify the trained solder joint defect detection model. If the verification fails, the training set is expanded to continue training. If the verification passes, then: The confidence level is calculated using the test set to test the verified solder joint defect detection model. If the test fails, the training set is expanded to continue training; if the test passes, the training is terminated.

5. The PCBA solder joint defect detection method based on improved RT-DETR according to claim 1, characterized in that: The step S4 is specifically as follows: The solder joint defect detection model that has passed the test is compressed using quantization technology and dynamic pruning technology, and the performance of the compressed solder joint defect detection model is verified. Actual PCBA solder joint images are collected, and drift compensation training is performed on the performance-verified solder joint defect detection model using each of the actual PCBA solder joint images. The solder joint defect detection model after drift compensation training is deployed using containerization technology.

6. A PCBA solder joint defect detection system based on improved RT-DETR, characterized by: Includes the following modules: A data set construction module is used to obtain a large number of historical PCBA solder joint images, pre-process and annotate each of the historical PCBA solder joint images, and then construct a data set; A solder joint defect detection model creation module is used to create a solder joint defect detection model based on the feature extraction layer, feature fusion layer, and prediction output layer, and set the loss function of the solder joint defect detection model. The feature extraction layer is built based on the backbone network of the RT-DETR model and is used to extract shallow features and deep features from the input PCBA solder joint image, and input the shallow features and deep features into the feature fusion layer. The feature fusion layer is constructed based on the local feature processing module, the global feature processing module and the feature interaction output module; The local feature processing module is constructed based on the input channel adjustment unit, the local enhancement unit, the structure optimization unit, and the residual connection unit. The input channel adjustment unit is used to perform channel compression and preliminary feature extraction on the input shallow and deep features through the first 1x1 convolution normalization activation layer to obtain the initial features. The local enhancement unit is used to enhance the local perception ability of the initial features and introduce nonlinear transformation through two stacked local enhancement sub-units to obtain enhanced initial features. The structure optimization unit is used to fuse the two-branch structure of the two local enhancement sub-units into a single 3x3 convolutional layer during the inference phase. The residual connection unit is used to fuse the enhanced initial features, shallow features, and deep features to obtain local features, which are then input into the global feature processing module. The global feature processing module is constructed based on the attention generation unit, the channel space mapping unit and the spatial compression unit; the attention generation unit is used to fuse the local features through global average pooling, global maximum pooling and Softmax function to obtain the fused attention feature; the channel space mapping unit is used to spatially map the local features through the second 1x1 convolution normalization activation layer to obtain the channel space mapping feature; the spatial compression unit is used to spatially compress the local features through the third 1x1 convolution normalization activation layer to obtain the spatial compression feature; The feature interaction output module is used to perform matrix multiplication on the fusion attention feature and the channel space mapping feature to obtain the spatial enhancement feature, and to perform matrix multiplication on the channel space mapping feature and the spatial compression feature to obtain the channel enhancement feature. The spatial enhancement feature is input into the 1x1 fourth convolution normalized activation layer to obtain the spatial attention map, and the channel enhancement feature is input into the unnormalized convolution activation layer to obtain the channel attention map. The spatial attention map and the channel attention map are element-wise multiplied to obtain the initial fusion feature, and the initial fusion feature is residually connected with the local feature to obtain the enhanced fusion feature and output it to the prediction output layer; a solder joint defect detection model training module, configured to divide the data set into a training set, a validation set, and a test set, train the solder joint defect detection model using the training set and a loss function, validate the trained solder joint defect detection model using the validation set, and test the validated solder joint defect detection model using the test set; A solder joint defect detection model deployment module is used to compress, drift compensate and deploy the solder joint defect detection model that has passed the test; The defect detection module is used to perform PCBA solder point defect detection by using the deployed solder point defect detection model.

7. A PCBA solder joint defect detection system based on improved RT-DETR according to claim 6, characterized in that: The dataset construction module is specifically used for: A large number of historical PCBA solder joint images with different lighting conditions, shooting angles, and PCBA models are acquired. Each of the historical PCBA solder joint images is preprocessed, including at least cropping, contrast adjustment, exposure adjustment, and size adjustment. The preprocessed historical PCBA solder joint images are labeled with defect types and defect locations using the labelimg annotation tool. A dataset is constructed based on the labeled historical PCBA solder joint images. The defect types include at least normal, kneeling foot, excessive solder, insufficient solder, and continuous solder.

8. The PCBA solder joint defect detection system based on improved RT-DETR according to claim 6, characterized in that: The prediction output layer is constructed based on the decoder of the RT-DETR model, and is used to map the enhanced fusion features into PCBA solder joint defect detection results carrying defect types and defect locations; The formula of the loss function is: ； in, Represents the loss value of the loss function; represents the defect classification loss; represents the bounding box regression loss; represents the bounding box overlap loss; represents feature consistency loss; 、 、 、 Both represent weight coefficients.

9. The PCBA solder joint defect detection system based on improved RT-DETR according to claim 6, characterized in that: The solder joint defect detection model training module is specifically: Dividing the data set into a training set, a validation set, and a test set in a ratio of 8:1:1 using a stratified sampling method, setting hyperparameters including at least a batch size, an optimizer, a learning rate, a weight decay rate, and a training cycle, and training a solder joint defect detection model using the training set and the hyperparameters until a loss value of the loss function is less than a preset loss threshold or a preset early stopping condition is met; The average precision mean, Giga floating point operations per second, parameter count, and frames per second are calculated using the validation set to verify the trained solder joint defect detection model. If the verification fails, the training set is expanded to continue training. If the verification passes, then: The confidence level is calculated using the test set to test the verified solder joint defect detection model. If the test fails, the training set is expanded to continue training; if the test passes, the training is terminated.

10. The PCBA solder joint defect detection system based on improved RT-DETR according to claim 6, characterized in that: The solder joint defect detection model deployment module is specifically used to: The solder joint defect detection model that has passed the test is compressed using quantization technology and dynamic pruning technology, and the performance of the compressed solder joint defect detection model is verified. Actual PCBA solder joint images are collected, and drift compensation training is performed on the performance-verified solder joint defect detection model using each of the actual PCBA solder joint images. The solder joint defect detection model after drift compensation training is deployed using containerization technology.

Images