Circuit board welding fault identification method and system based on machine vision

Abstract

The invention discloses a circuit board welding fault identification method and system based on machine vision, and relates to the technical field of machine vision and image processing. Performing spatial position weighting on the high-resolution features by using background suppression guide information from a large receptive field branch, so that the model maintains small target texture and edge information and reduces the response ratio of a non-welding spot region, thereby improving the detectability of micro welding spots and fine-grained defects under a complex background; according to the method, a large receptive field feature extraction branch is set to provide a welding spot contour and context relationship, and welding spot enhancement guide information is generated by a high-resolution branch to participate in a feature fusion process of a second branch, so that the second branch adopts differential fusion weights for a welding spot candidate region and a background region in a fusion stage; and interference of background factors such as light reflection, silk screen and board texture on large target regression and classification is reduced.

Description

Technical Field

[0001] This invention relates to the field of machine vision and image processing technology, and in particular to a method and system for identifying circuit board soldering faults based on machine vision. Background Technology

[0002] In modern electronics manufacturing, printed circuit boards (PCBs) and printed circuit boards (PCBAs) are continuously evolving towards miniaturization and high-density assembly. The same board often contains both micro-chip components such as 01005 (typically 0.4mm × 0.2mm in size) and large-size, multi-pin / multi-terminal packaged devices (such as QFPs, connectors, and fine-pitch array packages). Under machine vision imaging conditions such as automated optical inspection (AOI), fixed magnification or large field-of-view scanning / stitching methods are usually used to cover a large inspection area. Small-sized solder ends, pads, and their defect features may only appear as a few pixel details in the image, while exposed solder joints and outlines of large devices occupy a significantly larger pixel range, thus creating a cross-scale recognition requirement within the same inspection process. Some AOI systems also claim that their optical resolution can cover the 01005 size level.

[0003] To address the aforementioned cross-scale detection requirements, in addition to traditional algorithms based on rule thresholding, template matching, and geometric measurement, some existing technologies also employ deep learning-based target detection / segmentation methods. These methods enhance the representation capabilities of targets at different scales through feature pyramids, multi-scale feature fusion, attention mechanisms, or loss weighting.

[0004] However, under conditions of high-density mixed assembly and complex backgrounds (reflection, shadow, screen printing interference, device occlusion, etc.), deep networks usually need to make a trade-off between expanding the receptive field to obtain contextual information and maintaining high resolution to preserve the details of small solder joints / defects: if they rely too much on deep features, the features of small solder joints or fine-grained defects may be weakened; if they retain too much shallow high-resolution features, they may introduce background noise and affect the stable regression of the shape and position of larger targets, which may result in problems such as missed detection of small defects and reduced positioning accuracy of some large targets in the same system.

[0005] It should also be noted that for packages such as BGA where the bottom solder joints are obscured by the device, conventional optical AOI is usually difficult to perform direct visual inspection of the solder joints. In engineering practice, X-ray inspection and other methods are often used to supplement the inspection of the hidden solder joints. Summary of the Invention

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] This invention provides a method and system for identifying circuit board soldering faults based on machine vision, which solves the problem of simultaneous small missed detections and reduced positioning of large solder joints caused by densely mixed and cross-scale solder joints.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, embodiments of the present invention provide a method for identifying circuit board soldering faults based on machine vision, comprising: Step S1: Obtain an image of the surface of the circuit board under test, including exposed solder joints; Step S2: Input the surface image of the circuit board to be tested into the pre-trained solder joint defect detection model; Step S3: The solder joint defect detection model outputs the target location, confidence level, and corresponding defect category for each solder joint. The training of the solder joint defect detection model is based on a set of labeled sample images containing solder joints of different scales. The solder joint defect detection model includes a first feature extraction branch, a second feature extraction branch, a cross-branch guidance module, and a detection head. The first feature extraction branch is used to generate high-resolution features to characterize small-scale solder joint details, and the second feature extraction branch is used to generate large receptive field features to characterize large-scale solder joint contours and context. The cross-branch guidance module generates background suppression guidance information based on the intermediate features of the branch extracted from the second feature, and weights the corresponding features of the first branch according to spatial location to reduce the response of the background area that does not contain solder joints. At the same time, it generates solder joint enhancement guidance information based on the intermediate features of the branch extracted from the first feature and applies it to the feature fusion of the second branch to improve the separability of the solder joint candidate area from the background. The detection head fuses the guided features of the two branches to obtain a prediction result of the solder joint location and defect category.

[0009] As a preferred embodiment of the circuit board soldering fault identification method based on machine vision described in this invention, the background suppression guidance information is a spatial weight map of the same scale as the output features of the first feature extraction branch. The cross-branch guidance module obtains the spatial weight map by performing channel mapping on the intermediate features of the second branch and performing nonlinear normalization, and applies the spatial weight map to the features of the first branch to suppress the response of the background region.

[0010] As a preferred embodiment of the circuit board soldering fault identification method based on machine vision described in this invention, the solder joint enhancement guidance information is a solder joint candidate region indication map, the cross-branch guidance module generates the solder joint candidate region indication map based on the intermediate features of the branch extracted by the first feature, and assigns a higher fusion weight to the region corresponding to the indication map when the features of the second branch are fused.

[0011] As a preferred embodiment of the circuit board soldering fault identification method based on machine vision described in this invention, wherein: when training the solder joint defect detection model, the scale is divided according to the normalized area of ​​the labeled solder joint and the scale threshold, and the solder joint samples less than or equal to the scale threshold are regarded as small-scale samples, and the solder joint samples greater than the scale threshold are regarded as large-scale samples. When training the solder joint defect detection model, the scale division is based on the normalized area of ​​the labeled solder joint. The denominator of the normalized area is fixedly selected from the whole image region, the candidate region corresponding to the solder joint, or the device region to which the solder joint belongs during the training configuration phase. The generation rule of the scale threshold is fixedly selected from the validation set scanning rule, the generation rule based on the data distribution quantile, or the generation rule based on binary clustering during the training configuration phase. For solder joint samples within the preset boundary range of the scale threshold, they are merged into small-scale samples according to the single-attribution rule, or simultaneously included in the two-branch training statistics according to the continuous weight allocation rule to smooth the sample attribution at the threshold boundary.

[0012] As a preferred embodiment of the circuit board soldering fault identification method based on machine vision described in this invention, the loss function used to train the solder joint defect detection model consists of a first branch loss, a second branch loss, and a consistency loss; wherein during the training phase, a first branch prediction head is set at the first branch output to generate prediction results for small-scale samples and the first branch loss is calculated, and a second branch prediction head is set at the second branch output to generate prediction results for large-scale samples and the second branch loss is calculated.

[0013] As a preferred embodiment of the machine vision-based circuit board soldering fault identification method of the present invention, the consistency loss is used to constrain the consistency of the prediction results generated by the same solder joint in the first feature extraction branch and the second feature extraction branch. The prediction results include the defect category probability distribution and normalized location information. The consistency of the defect category probability distribution is achieved by using symmetric divergence or cross-entropy alignment measure after temperature smoothing of the category outputs of the two branches. The consistency of the normalized location information is achieved by using the distance between the center point and the width and height or the cross-union ratio regression consistency measure. In the training phase, confidence gating and weighted scheduling that increases with each round are combined to suppress low confidence alignment noise.

[0014] As a preferred embodiment of the machine vision-based circuit board soldering fault identification method of the present invention, wherein: the first feature extraction branch includes a shallow convolution and feature refinement layer for maintaining spatial resolution, the second feature extraction branch includes a downsampling layer and a context aggregation layer for expanding the receptive field, and the detection head is a one-stage detection head or a two-stage detection head.

[0015] Secondly, the present invention provides a circuit board soldering fault identification system based on machine vision, including, An image acquisition unit is used to acquire images of the surface of the circuit board under test, including exposed solder joints. The defect identification unit is used to load the solder joint defect detection model and perform inference on the surface image, and output the target position, confidence level and defect category of each solder joint; The model training unit is used to acquire a set of labeled sample images containing weld points of different scales and to train or iteratively update the weld point defect detection model according to training constraints.

[0016] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any step of the machine vision-based circuit board soldering fault identification method described in the first aspect of the present invention.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the machine vision-based circuit board soldering fault identification method as described in the first aspect of the present invention.

[0018] Through the above technical solution, the present invention can achieve at least the following beneficial effects: 1) To address the problem that small-scale solder joint details are easily diluted by deep features, leading to missed detection, this invention uses a high-resolution feature extraction branch to specifically preserve spatial detail representations and utilizes background suppression guidance information from a large receptive field branch to spatially weight high-resolution features. This allows the model to maintain the texture and edge information of small targets while reducing the response proportion of non-solder joint areas, thereby improving the detectability of tiny solder joints and fine-grained defects in complex backgrounds.

[0019] 2) To address the issue that background noise introduced by high-resolution features affects the stability of large-scale solder joint localization and classification, this invention sets up a large receptive field feature extraction branch to provide solder joint contours and contextual relationships. The high-resolution branch generates solder joint enhancement guidance information to participate in the feature fusion process of the second branch. This allows the second branch to apply differentiated fusion weights to the solder joint candidate region and the background region during the fusion stage, reducing the interference of background factors such as reflection, silkscreen printing, and board texture on the regression and classification of large targets.

[0020] 3) To address the problem that the same feature stream cannot take into account both details and context, and that the fusion trade-off leads to losses at both ends, this invention adopts a collaborative mechanism of dual-branch parallel representation and cross-branch bidirectional guidance. This decouples detail fidelity and context integrity into different feature paths. Then, the guidance module redistributes the focus of attention at the feature level, so that cross-scale targets can obtain more matching feature support under the same field of view, thereby structurally alleviating cross-scale conflicts.

[0021] 4) To address the issues of uneven distribution of samples across scales and imbalance in the attention given to different scales during training, this invention establishes a unified and comparable scale metric using normalized area and solidifies the normalization caliber and threshold generation rules during the training configuration phase, so that samples are stably assigned to different branches of training objectives according to scale. At the same time, a smooth assignment strategy for boundary samples is introduced to reduce sample jitter and gradient oscillation caused by hard thresholds, so that small-scale and large-scale samples can obtain more stable coverage and optimization during training.

[0022] 5) To address the issue of inconsistent outputs from two branches to the same solder joint, which can lead to duplicate boxes, category jumps, or pairing errors, this invention introduces consistency constraints to make the probability distribution of defect categories and normalized position information of the two branches on the same solder joint more consistent. It also combines confidence gating to suppress alignment noise caused by low-confidence samples, and gradually strengthens the consistency constraint through weighted scheduling that increases with each training round, thereby improving the output consistency and result availability during two-branch collaborative reasoning.

[0023] 6) To address the issue of scale metric drift caused by changes in camera magnification, cropping ratio, or board proportion, which in turn affects cross-batch generalization, this invention provides multiple normalized area calibers and fixes their selection in the training configuration, so that scale division and branch responsibilities match the acquisition conditions; without changing the dual-branch architecture, by using a unified scale metric and a reproducible threshold determination strategy, the model can more easily maintain the consistency of scale division and the transferability of the detection process under different production line acquisition conditions. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0025] Figure 1 This is a flowchart of the circuit board soldering fault identification method in the embodiment.

[0026] Figure 2 This is a framework diagram of the circuit board soldering fault identification system in the embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] Example 1: like Figure 1 As shown, this application proposes a machine vision-based method for identifying circuit board soldering faults, including the following steps: Step S1: Acquire an image of the circuit board surface containing exposed solder joints. In this embodiment, exposed solder joints refer to solder joint areas that can be directly imaged in the image of the circuit board surface. Their pixel appearance includes solder, pads, and local substrate texture information adjacent to the solder joints, so that the location and defect category can be determined from the image of the circuit board surface without introducing additional imaging aperture. The image of the circuit board surface is acquired by the image acquisition unit described later. During acquisition, at least the lens magnification and whole-image cropping ratio related to the training configuration are recorded to ensure that the scale measurement aperture used in the training phase is consistent with the actual acquisition aperture, so as to avoid the relative proportion of the same solder joint appearing under different acquisition settings drifting.

[0030] Step S2: Input the surface image of the circuit board to be tested into the pre-trained solder joint defect detection model; Step S3: The solder joint defect detection model outputs the target location, confidence level, and corresponding defect category for each solder joint. Specifically, the target location is used to characterize the spatial position of the solder joint in the coordinate system of the network input image. In the subsequent consistency constraints of this embodiment, normalized position information is used for alignment. The confidence level is the degree of credibility of the detection head's prediction result for the solder joint. Its numerical range is usually limited to 0 to 1 so as to be directly combined with gating and weighted scheduling. The defect category is one of the predefined defect category sets. The total number of defect categories and the category code are fixed in the training configuration stage and are consistent in the sample labeling and model output to ensure that the probability distribution of defect categories has consistent semantics in the training and inference stages.

[0031] The solder joint defect detection model is trained based on a set of labeled sample images containing solder joints at different scales. The model includes a first feature extraction branch, a second feature extraction branch, a cross-branch guidance module, and a detection head. For example, each labeled solder joint in the labeled sample image set corresponds to at least one labeled region for localization and one defect category label. The labeled region can be given by a solder joint bounding box or a solder joint segmentation mask, so that the pixel area of ​​the solder joint can be directly obtained when dividing the scale. When using a solder joint bounding box, the pixel area of ​​the solder joint can be approximately obtained according to the pixel area covered by the bounding box. When using a solder joint segmentation mask, the pixel area of ​​the solder joint can be obtained according to the pixel area within the mask. To ensure coverage of samples at different scales, the sample image set can be sampled and checked according to the statistical distribution of normalized area during the training configuration phase, so that both small-scale and large-scale samples have a sufficient number to participate in the two-branch training and consistency constraints.

[0032] The first feature extraction branch generates high-resolution features to characterize small-scale solder joint details, while the second feature extraction branch generates large-receptive-field features to characterize large-scale solder joint contours and context. Similarly, high-resolution features refer to feature representations that retain more pixel details in the spatial dimension, ensuring that the edges, textures, and local defect appearances of small-scale solder joints are not lost due to oversampling. Large-receptive-field features refer to feature representations that expand the effective receptive range through downsampling layers and context aggregation layers during feature extraction, ensuring that the overall contour, neighborhood context, and structural differences from the background of large-scale solder joints are stably encoded. The two types of features maintain their respective emphases before the detection head is fused, thus providing a basis for subsequent cross-branch guided information complementarity.

[0033] The cross-branch guidance module generates background suppression guidance information based on the intermediate features of the second branch, and weights the corresponding features of the first branch according to spatial location to reduce the response of background regions that do not contain solder joints. At the same time, it generates solder joint enhancement guidance information based on the intermediate features of the first branch and applies it to the feature fusion of the second branch to improve the separability of solder joint candidate regions from the background. Further, the intermediate features are the intermediate layer output feature maps generated by the corresponding feature extraction branches during the forward propagation process. The selected layer is fixed during the training configuration phase to ensure the semantic and spatial scale stability of the guidance information. Channel mapping is used to compress or reorganize the channel dimension information of the intermediate features into the channel shape required to generate guidance information. In engineering implementation, channel number mapping can be completed through convolutional layers while keeping the spatial size unchanged. Nonlinear normalization is used to compress the numerical range of guidance information to a range that is easy to weight. In engineering implementation, Sigmoid or Softmax operators can be selected to make the values ​​of the spatial weight map or candidate region indicator map fall between 0 and 1, so that they can be directly used to perform spatial location-based weighted suppression or weighted enhancement on the feature map.

[0034] The detection head fuses the guided features from the two branches to obtain predictions of the weld location and defect type. In this embodiment, the background suppression guidance information is a spatial weight map of the same scale as the first branch features. The cross-branch guidance module obtains the spatial weight map by performing channel mapping on the intermediate features of the second branch and nonlinear normalization, and applies the spatial weight map to the first branch features to suppress the background region response. Optionally, the spatial weight map can have a nearly uniform initial distribution at the beginning of training, so that the first feature extraction branch can still fully learn the solder joint details in the early stage. Subsequently, with training iterations, the cross-branch guidance module gradually forms the suppression difference of the background region. The spatial weight map and the output features of the first feature extraction branch are of the same scale to ensure that they correspond pixel by pixel when weighted by spatial position. If there is a difference in spatial size between the two, the spatial size can be aligned by upsampling or downsampling without changing the channel mapping and nonlinear normalization definition. The value of the spatial weight map is generally not taken as an extreme value of strictly 0 or strictly 1 to avoid the feature response being completely truncated or completely released, which would cause training instability.

[0035] In this embodiment, the solder joint enhancement guidance information is a solder joint candidate region indication map. The cross-branch guidance module generates a solder joint candidate region indication map based on the intermediate features of the first branch, and assigns a higher fusion weight to the region corresponding to the indication map when the features of the second branch are fused. In this embodiment, the solder joint candidate region indicator map is used to characterize the spatial region that the second feature extraction branch should focus on more during the feature fusion stage. Its source is the spatial distribution of the intermediate features of the first feature extraction branch obtained by channel mapping and nonlinear normalization. Higher fusion weight can be achieved by using the indicator map as a fusion coefficient to participate in weighted summation or element-wise weighting, thereby increasing the effective response of the second feature extraction branch to the solder joint candidate region and decreasing the effective response to the background region during fusion without introducing a new network structure. The numerical range of the indicator map can also be limited to 0 to 1, and the spatial alignment aperture can be consistent with the fusion input of the subsequent detection head.

[0036] In this embodiment, when training the solder joint defect detection model, the scale is divided according to the normalized area of ​​the labeled solder joint and the scale threshold. Solder joint samples that are less than or equal to the scale threshold are regarded as small-scale samples, and solder joint samples that are greater than the scale threshold are regarded as large-scale samples. When training the solder joint defect detection model, the scale division is completed based on the normalized area of ​​the labeled solder joint. The denominator of the normalized area is fixedly selected from the whole image region, the candidate region corresponding to the solder joint, or the device region to which the solder joint belongs during the training configuration phase. The scale threshold generation rule is fixedly selected from the validation set scanning rule, the data distribution quantile generation rule, or the binary clustering generation rule during the training configuration phase. For solder joint samples near the threshold boundary, they are merged into small-scale samples according to the single-attribution rule, or simultaneously included in the two-branch training statistics according to the continuous weight allocation rule to smooth the sample attribution at the threshold boundary. In one implementation, during the training phase, scale division is achieved by calculating the normalized area for each labeled solder joint and then comparing it with a scale threshold. Specifically, the solder joint area data is directly extracted from the labeled area of ​​the labeled solder joint, and the extraction method is consistent with the labeling format. When the labeled area is a solder joint segmentation mask, the solder joint pixel area is taken as the area of ​​the pixel area covered by the mask. When the labeled area is a solder joint label box, the solder joint pixel area is taken as an approximation as the area of ​​the pixel area covered by the label box. The normalized area is the ratio of the solder joint pixel area to the selected denominator pixel area. Therefore, its value range is usually between 0 and 1 in engineering and is comparable in the same training configuration. To ensure the reproducibility of scale division, the selection of the normalized area and the scale threshold generation rules are both fixed during the training configuration phase and recorded in the training log so as to reproduce the experimental conditions during iterative updates.

[0037] The denominator selection for the normalized area is related to the data acquisition method. Without changing the dual-branch architecture, any of the following calibers can be used, and one caliber should be consistently selected in the training configuration to maintain consistency in the annotation scale. For example, when the overall field of view and board proportion are stable, the overall image caliber ensures that the normalized area is highly consistent with the overall image cropping ratio, facilitating scale comparison under the same acquisition settings. When the camera magnification or overall image cropping ratio changes, the local candidate region caliber, by limiting the denominator to the pixel area of ​​the candidate region corresponding to the solder joint, can mitigate the impact of magnification changes. The relative proportion shift caused by this; when there is a large difference in device size within the same batch, the device region aperture can be limited to the pixel area of ​​the device region to which the solder joint belongs by limiting the denominator, so that the solder joint scale measurement on different devices can have better cross-device consistency; if the denominator source selected in the training configuration cannot be located from the device label box, device segmentation mask or device reference number on individual samples, the aperture in the training configuration can be kept unchanged and the sample can be marked as missing measurement, or it can be reverted to the aperture of the whole image region to ensure that the normalized area can still be calculated, thereby avoiding the interruption of the scale division process.

[0038] When the normalized area is defined according to the whole map caliber, let: (1) In equation (1), Indicates the first The first sample image The normalized area of ​​each labeled solder joint relative to the entire image. Indicates the first The first sample image The pixel area of ​​each labeled solder joint Indicates the first The total pixel area of ​​a sample image; this aperture is suitable for acquisition scenarios where the overall field of view and the proportion of the board are relatively stable. When the normalized area is defined according to the ROI, let: (2) In equation (2), Indicates the first The first sample image The normalized area of ​​each labeled solder joint relative to its corresponding ROI. Indicates the relationship with the first The first sample image The ROI pixel area corresponding to each solder joint can be determined by the device outer rectangle, the pad array outer frame, or the detection candidate region; this aperture is used to reduce the offset of scale division caused by different camera magnification and different whole image cropping ratio. When the normalized area is defined according to the device region aperture, let: (3) In equation (3), Indicates the first The first sample image The normalized area of ​​each labeled solder joint relative to the region of its component. Indicates the first The first sample image The pixel area of ​​the device region to which each solder joint belongs is obtained by the device label box, device segmentation mask or device reference number; this aperture is used to adapt to the scenario of board cards with large differences in device size in the same batch. After selecting the normalized area caliber, it can be uniformly denoted as... and according to scale threshold Complete the partition: when When the corresponding solder joint samples are classified into small-scale samples, The corresponding solder joint samples are then grouped into large-scale samples; among them, This indicates the first caliber obtained using the selected aperture. The first sample image Normalized area of ​​each labeled solder joint This represents the scale threshold used to distinguish between small and large scales; Scale threshold The value of can be selected using one of the following strategies, and its generation rules can be fixed in the training configuration to reproduce the experimental results: Experience threshold: Based on the lens magnification and typical proportion of solder joints in the existing production line, several candidate thresholds are scanned on the validation set to ensure that the ratio of small-scale sample quantity to large-scale sample quantity falls within the preset range, so as to reduce the risk of missed detection caused by unilateral sample sparsity. Multiple threshold levels: Introducing a set of multiple threshold levels The scale is segmented, and then the intermediate scale segments are categorized by proximity or simultaneously included in the mapping back to the two-branch training objectives. This is used to cover data distributions with large continuous scale variations. Indicates the number of threshold levels; Quantile threshold: When it is desired that the threshold adapts to the data distribution, let: (4) In equation (4), Represents a set Pick Statistical calculation of quantiles, This represents the quantile coefficient, with a range of values ​​of 100. This strategy is used to maintain the stability of the proportion of small-scale samples across different batches of data. This is the normalized area set of all labeled weld points in the training sample image set; Clustering threshold: when When it exhibits a bimodal or multimodal distribution, it can be used for... Perform K-means clustering with a cluster size of 2 to obtain the centers of the two classes. and And order: (5) In equation (5), This represents the center value of a smaller-scale cluster. This represents the center value of a larger-scale cluster; this strategy is used to determine the boundary point by utilizing the data's own structure, reducing fluctuations caused by manually selecting thresholds. Furthermore, the validation set scanning rule can traverse several candidate scale thresholds, and is constrained by the proportion of small-scale and large-scale samples falling within a preset range, so that the training samples of the two branches do not become extremely unbalanced; in the quantile generation rule, the quantile coefficient q is usually taken near the median of close to 0.5 to obtain a relatively balanced sample division, and can be fine-tuned according to the validation set performance within a range greater than 0 and less than 1, avoiding taking values ​​close to 0 or close to 1, which would result in too few samples in one branch and insufficient training; in the binary clustering generation rule, the number of K-means clusters is fixed at 2 to match the purpose of dividing the small-scale and large-scale classes, the clustering input is the normalized area set of all labeled welding points in the training sample image set, and the two class center values ​​output by the clustering are used to give a stable boundary point, and this boundary point is fixed in the training configuration stage for reproducibility; in the multi-level threshold set, the number of threshold levels L can be taken in a small integer range of 2 to 4 according to the data distribution complexity, so as to cover the distribution with large continuous scale changes, while avoiding the excessive number of boundary samples caused by the excessive number of threshold levels, which would increase the statistical fluctuation of training.

[0039] The rules for assigning boundary samples can be set in accordance with the training objective: Single-attribution rule: will satisfy The samples are incorporated into the small-scale samples so that the high-resolution branches cover more boundary detail samples, reducing the probability that small defects at the boundary are submerged by low-resolution features. Dual Attribution Rule: Samples near the boundary are simultaneously included in the training statistics of both branches, and their contribution is controlled by continuous weights. Let: (6) In equation (6), Indicates the first The first sample image The weights of each solder joint sample in the first branch training This indicates the weight of the sample during training in the second branch. This represents the Sigmoid function. This represents the boundary smoothing coefficient, used to control the range of transition from single-assignment to double-assignment; this rule is used to reduce training instability caused by hard partitioning of samples near the threshold. Optionally, boundary smoothing coefficient The width used to control the transition from single to double assignment of boundary samples has engineering significance in providing a smooth interval on the numerical axis of the normalized area, so that samples falling into this interval will not cause abrupt changes between the two branches due to hard thresholding; when the normalized area varies from 0 to 1, The value can be in the range of 0.01 to 0.10 to cover the boundary density of different data distributions. The default value can be 0.03 to 0.05 to balance the smoothing effect and the sample discrimination. τ should not be too small to avoid still being an approximate hard split, nor should it be too large to avoid too many samples being regarded as boundary samples and weakening the division of labor between the two branches.

[0040] During the training phase, for each labeled solder joint in each labeled sample image, the solder joint area data is extracted based on the pixel area of ​​the labeled region, and the source of the denominator is determined in combination with the normalization caliber selected in the training configuration. When the whole image caliber is used, the normalized area index relative to the whole image is obtained according to equation (1); when the local candidate region caliber is used, the normalized area index relative to the candidate region is obtained according to equation (2); when the device region caliber is used, the normalized area index relative to the device region is obtained according to equation (3). Then, the normalized area index is compared with the scale threshold to complete the division between small-scale samples and large-scale samples. The data can be obtained by calculating the data distribution quantiles according to Equation (4), or by calculating the binary clustering according to Equation (5). Alternatively, the candidate threshold can be scanned on the validation set to meet the sample ratio constraint. When a sample falls near the threshold boundary, it can be merged into the small-scale sample according to the single-attribution rule, or the continuous contribution of the two branches can be calculated according to Equation (6) and simultaneously included in the training statistics of the two branches. The output scale division result will serve as the basis for allocating samples to different branches, calculating the sample ratio, and allocating the training contribution to the boundary samples during the training phase, thereby matching the coverage of the high-resolution branch on the detail samples and the coverage of the large receptive field branch on the contour samples.

[0041] Specifically, the scale division process in the above implementation establishes a unified and comparable scale metric around the relative area of ​​the solder joints, and then uses a scale boundary to divide the samples into two categories, which are used to match the feature expression emphasis of different branches. The denominator of the normalized area can be set according to the whole image, local area, or device area. Different apertures correspond to the scale stability requirements under different acquisition conditions: the whole image aperture is adapted to the scene with a fixed field of view, and the local or device aperture is used to reduce the offset caused by the difference in magnification and cropping ratio. The value of the scale boundary can be determined by empirical rules based on validation set scanning, or by automatically deriving it from the data distribution using statistical quantiles or clustering methods, so that different batches of data are consistent in terms of sample proportion and scale coverage. For critical samples, rules can be set for fixed attribution or joint inclusion in both branches. The contribution of critical samples between the two branches can be adjusted by continuously changing weights, thereby alleviating the training fluctuations caused by hard thresholds and enabling small-scale details and large-scale contour samples to obtain more stable coverage during training. In this embodiment, the loss function used to train the weld joint defect detection model consists of a first branch loss, a second branch loss, and a consistency loss; wherein during the training phase, a first branch prediction head is set at the output of the first branch to generate prediction results for small-scale samples and calculate the first branch loss, and a second branch prediction head is set at the output of the second branch to generate prediction results for large-scale samples and calculate the second branch loss. In this embodiment, consistency loss is used to constrain the consistency of prediction results generated by different branches for the same solder joint. The prediction results include at least the defect category probability distribution and normalized location information. The consistency loss is used to constrain the consistency between the probability distribution of defect categories and the normalized location information generated by different branches for the same solder joint. The consistency of the probability distribution of defect categories is achieved by using symmetric divergence or cross-entropy alignment measures after temperature smoothing of the output of the two branches. The consistency of the normalized location information is achieved by using a distance measure based on the center point and width and height or a regression consistency measure based on the cross-union ratio. During the training phase, the consistency loss is combined with confidence gating to suppress alignment noise of low-confidence samples, and a weighted scheduling that increases with the training rounds is used to gradually increase the consistency constraint strength. In one implementation, during the training phase, a consistency loss is introduced onto the sample set where both branches simultaneously obtain prediction results for the same labeled weld point, aligning the defect category probability distribution of the two branches with the normalized location information for that weld point; let this sample set be... For any solder joint sample The first and second branches output the category logit vectors respectively. , With confidence level , and output the normalized position vector. , ;in, This represents the set of solder joint samples that participate in the consistency constraints. Indicates the solder joint sample index. Indicates the first branch on the sample The defect category logit vector, Indicates the second branch on the sample The defect category logit vector, Indicates the first branch on the sample The prediction confidence level Indicates the second branch on the sample The prediction confidence level Indicates the first branch on the sample Normalized position information vector, Indicates the second branch on the sample The normalized position information vector; Specifically, the sample set is the set of labeled solder joints that can obtain valid prediction entries simultaneously from the first branch prediction head and the second branch prediction head during the training phase. The valid prediction entries are determined by the target position and confidence level output by the detection head. When a labeled solder joint generates a valid prediction entry only on one branch, the solder joint can be excluded from the consistency loss calculation to avoid noise caused by the lack of alignment objects. When the same labeled solder joint generates multiple candidate prediction entries on both branches, a pair of prediction entries can be selected to form an alignment sample according to the principle of the closest position or the greatest overlap, so that the consistency constraint is concentrated on the prediction result that is most likely to correspond to the same solder joint, thereby reducing the interference of duplicate boxes on consistency alignment.

[0042] The specific steps for constraining the consistency loss of defect category probability distribution and normalized location information include: A. Defect Category Probability Distribution Consistency Measurement (with Temperature Coefficient): When the class distribution is generated by logit, applying a temperature coefficient to the two branches of logit yields a smoothed probability distribution: (7) In equation (7), Indicates the first Branch pairs of samples The probability distribution vector of the defect category. Represents the Softmax operator. This represents the temperature coefficient of the category distribution, with a value range of [value missing]. to , The distribution becomes more gradual as the size increases. Indicates a branch index. These correspond to the first branch and the second branch, respectively. Similarly, the dimension of the category logit vector is the same as the total number of defect categories C, and its value is the unnormalized score of the detection head for each defect category; temperature coefficient Used to adjust the smoothness of the Softmax output distribution. The default value is around 2.0 to reduce gradient fluctuations caused by an overly sharp distribution, and can be adjusted in the range of 1.0 to 4.0 depending on the performance of the validation set. It is not advisable to take values ​​that are too small, close to or below 1.0, to avoid the distribution being too sharp and amplifying small differences between branches. It is also not advisable to take values ​​that are too large, to avoid the distribution being too flat and reducing the class discrimination, thereby affecting the effectiveness of consistency alignment.

[0043] Consistency of category distributions can be measured using symmetric KL divergence: (8) In equation (8), Indicates sample Class distribution consistency loss, Indicates KL divergence; Will Write it in the form of summation by category: (9) In equation (9), Indicates branch For the sample The category probability distribution vector, Indicates branch For the sample The category probability distribution vector, express In the The probability components of each defect category express In the The probability components of each defect category Indicates the total number of defect categories. Represents the natural logarithm operator. and Represents a branch index and satisfies ; In the same position, JS divergence can also be used to obtain natural symmetry. JS divergence reduces the numerical fluctuation caused by extreme probabilities by introducing intermediate distributions. It is a distribution alignment metric similar to symmetric KL. When cross-entropy alignment is used, one branch distribution can be used as a soft label to align another branch. Its form is mathematically equivalent to the KL term in the above formula, except for the constant term. In training implementation, it can be selected according to computational convenience. B. Normalized location information consistency measure (center point and width / height, L1 or IoU family) Normalized location information can be represented by the center point and width and height as follows: (10) In equation (10), Indicates the first Branch pairs of samples The horizontally normalized coordinates of the center point of the prediction box. Indicates the first Branch pairs of samples The vertical normalized coordinates of the center point of the prediction box Indicates the first Branch pairs of samples The normalized value of the prediction box width. Indicates the first Branch pairs of samples The normalized value of the predicted bounding box height; When the normalized reference is the network input image size At that time, the normalized coordinates can be obtained using the ratio of the number of pixels to the input size: (11) In equation (11), Indicates the first Branch pairs of samples The horizontal pixel coordinates of the center point of the prediction bounding box Indicates the first Branch pairs of samples The vertical pixel coordinates of the center point of the prediction box. Indicates the first Branch pairs of samples The pixel value of the predicted bounding box width. Indicates the first Branch pairs of samples The pixel value of the predicted bounding box height. The pixel value representing the width of the network input image. The pixel value representing the height of the network input image; Positional consistency can be achieved using the center point and width / height. measure: (12) In equation (12), Indicates sample Location consistency loss, Representing vectors Norm; When it is desired that location consistency is more sensitive to overlapping regions, it can be replaced with IoU family consistency regression, an example of the DIoU form: (13) In equation (13), This represents the intersection-union operator for two frames. Indicates by Restore the first branch prediction box. Indicates by The second branch prediction box was restored. express and Euclidean distance between the center points This represents the diagonal length of the smallest bounding box of the two frames; C. Weighted combination and training stability techniques (confidence gating and weight range) To reduce alignment noise caused by low-confidence predictions, a confidence gating coefficient can be introduced. And weighted for consistency loss: (14) In equation (14), Indicates sample Consistency gating coefficient, This represents the Sigmoid operator. This represents the confidence level gating threshold, with a value range of [value range missing]. to , This represents the gating smoothing coefficient, with a value range of [value missing]. to ; Furthermore, the confidence gating threshold The confidence interval used to determine the primary role of consistency constraints is set to 0.5 by default to strike a balance between reliable predictions and the number of samples covered, and can be adjusted within the range of 0.3 to 0.7 in conjunction with the validation set; the gating smoothing coefficient... Used to control the transition slope of the gating from low to high, the default value is about 0.10, and it can be adjusted in the range of 0.05 to 0.20 to balance gating stability and sensitivity; It is not advisable to set the value close to 0 to avoid a large number of low-confidence predictions participating in alignment and introducing noise, nor is it advisable to set the value close to 1 to avoid overly strict gating and resulting in too few consistency constraint samples.

[0044] The consistency loss can be weighted by summing class consistency and location consistency, and then normalized over the sample size: (15) In equation (15), Indicates consistency loss. Represents a set The number of samples in This represents the category consistency weight, with a value range of [value range missing]. to , This represents the positional consistency weight, with a value range of [value range missing]. to ; Optionally, the category consistency weight and position consistency weight are used to balance the relative contributions of the two alignment objectives. By default, the category consistency weight can be set to about 0.5 and the position consistency weight to about 1.0 so that position alignment slightly dominates and stabilizes the spatial convergence of the box. Subsequently, the values ​​can be fine-tuned according to the performance on the validation set within the range given in the original text. When the defect category is more easily confused while the position is relatively stable, the value of the category consistency weight can be appropriately increased. When the dense solder joints make the position regression more sensitive, the value of the position consistency weight can be appropriately increased. Both should not be set to near the upper limit at the same time to avoid the consistency term suppressing the complementary learning of the branches too much.

[0045] When introducing a consistency term early in training amplifies random prediction fluctuations, it can be adjusted accordingly. and Apply an increasing coefficient that grows with each training epoch to gradually strengthen the consistency constraint after the branch has a basically usable output: (16) In equation (16), Indicates the first Class consistency weights for each training round. Indicates the first Position consistency weights for each training round This indicates the upper limit of the category consistency weight. This indicates the upper limit of the positional consistency weight. This represents the weighting increase coefficient. Indicates the training round index. This indicates the end of the round when the weight increases; the value range is... to In this embodiment, the round ends when the weight increases. Used to specify the transition length of consistency constraints from weak to strong. The default value is around 5, so that the two branches can be gradually strengthened to align after they have basic usable output. The value can be adjusted within the range of 3 to 10. When there are large fluctuations in the early stage of training, a larger value can be used to prolong the slow start. When the early stage of training is relatively stable, a smaller value can be used to form alignment faster. The engineering significance of this scheduling method is to avoid the consistency term dominating the gradient in the early stage of training, thereby improving the convergence stability and branch coordination effect of the training process without changing the dual-branch and detector head structure.

[0046] During the training phase, consistency constraints are constructed on the sample set where both branches can simultaneously obtain prediction results at the same labeled solder joint. Alignment is performed based on the class scores and confidence levels of the two branches and the normalized position prediction information. When aligning the class distribution, the class scores of the two branches are first smoothed by temperature according to Equation (7) to obtain the probability distribution. Then, the class consistency loss in the form of symmetric divergence is calculated according to Equation (8), and the divergence summation by class dimension is completed according to Equation (9). When aligning the position, the normalized position representation is organized according to Equation (10), and the prediction box is normalized from the number of pixels to the position information under the input size reference according to Equation (11). Then, it can be aligned according to Equation (12). The consistency of the distance between the center point and the width and height can be calculated using Equation (13), and the regression consistency based on the overlap relationship and the center distance term can also be calculated using Equation (14). To reduce the alignment noise caused by low confidence prediction, the gating coefficient is calculated from the confidence of the two branches using Equation (14), and the consistency loss is formed by weighted combination of category consistency and position consistency using Equation (15). When the output fluctuates greatly in the early stage of training, the combined weight of the consistency term is adjusted according to Equation (16) with increasing scheduling as the training rounds progress. The consistency loss of the output will participate in backpropagation together with the training loss of each of the two branches, so that the category judgment and position prediction of the two branches at the same welding point tend to be consistent.

[0047] Specifically, consistency loss is used to make two feature branches give similar classification judgments and position predictions for the same solder joint, thereby reducing duplicate boxes, category jumps or missing pairings caused by differences in output between branches. Category consistency is achieved through distribution alignment, where the category outputs of the two branches are treated as soft labels that mutually constrain each other, making the probability distribution of defect categories more consistent on the same samples. To reduce gradient fluctuations caused by sharp, overconfident distributions, a temperature coefficient can be introduced to smooth the probability distribution. Positional consistency is achieved by aligning the center point with the width and height, which can be done using a distance metric based on vector difference or a regression metric related to the overlapping area, making the prediction boxes of the two branches converge in space. To reduce the interference of low-confidence predictions on training, a gating coefficient can be constructed based on the confidence of the two branches, making the consistency constraint more concentrated on reliable samples. When the consistency term is easily affected by random outputs in the early stages of training, a weighting strategy that increases with each round can be used to gradually increase the constraint strength. The above settings do not change the original dual-branch and detector head structure, but only enhance the collaboration between the two branches at the training objective level. In this embodiment, the first feature extraction branch includes a shallow convolutional layer and a feature refinement layer to maintain spatial resolution, the second feature extraction branch includes a downsampling layer and a context aggregation layer to expand the receptive field, and the detection head is a one-stage detection head or a two-stage detection head.

[0048] Example 2: Based on Example 1, such as Figure 2 As shown, this application also proposes a machine vision-based circuit board soldering fault identification system, including: An image acquisition unit is used to acquire images of the surface of the circuit board under test, including exposed solder joints. The defect identification unit is used to load the solder joint defect detection model and perform inference on the surface image, and output the target position, confidence level and defect category of each solder joint; wherein the solder joint defect detection model is a neural network model trained by the method of Example 1. The model training unit is used to acquire a set of labeled sample images containing weld points of different scales and to train or iteratively update the weld point defect detection model according to the training constraints of Example 1.

[0049] This embodiment also provides a computer device, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the circuit board soldering fault identification method and system based on machine vision as proposed in the above embodiment.

[0050] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0051] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the machine vision-based circuit board soldering fault identification method and system proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0053] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments. For example, all the embodiments above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for identifying a soldering defect of a circuit board based on machine vision, characterized by, The method comprises the following steps: Step S1, acquiring a surface image of a circuit board to be tested containing exposed solder joints; Step S2, inputting the surface image of the circuit board to be tested into a pre-trained solder joint defect detection model; Step S3, outputting the target position, confidence and corresponding defect category of each solder joint by the solder joint defect detection model; The training of the solder joint defect detection model is based on a set of labeled sample images containing solder joints of different scales, and the solder joint defect detection model comprises a first feature extraction branch, a second feature extraction branch, a cross-branch guidance module and a detection head; The first feature extraction branch is used to generate high-resolution features to represent the details of small-scale solder joints, and the second feature extraction branch is used to generate large receptive field features to represent the outlines and context of large-scale solder joints; The cross-branch guidance module generates background suppression guidance information based on the intermediate features of the second feature extraction branch, and weights the corresponding features of the first branch according to the spatial position to reduce the response of the background area not containing the solder joint, while generating solder joint enhancement guidance information based on the intermediate features of the first feature extraction branch and acting on the feature fusion of the second branch to improve the separability of the solder joint candidate area and the background; The detection head fuses the guided features of the two branches to obtain the prediction results of the solder joint position and the defect category.

2. The machine vision-based circuit board solder joint failure identification method of claim 1, wherein, The background suppression guidance information is a spatial weight map of the same scale as the output features of the first feature extraction branch, and the cross-branch guidance module obtains the spatial weight map by channel mapping and nonlinear normalization of the intermediate features of the second branch, and applies the spatial weight map to the first branch features to suppress the background area response.

3. The machine vision-based circuit board solder joint failure identification method of claim 1, wherein, The solder joint enhancement guidance information is a solder joint candidate area indication map, and the cross-branch guidance module generates the solder joint candidate area indication map based on the intermediate features of the first feature extraction branch, and gives a higher fusion weight to the area corresponding to the indication map during the feature fusion of the second branch.

4. The machine vision-based circuit board solder joint failure identification method of claim 1, wherein, During the training of the solder joint defect detection model, the labeled solder joints are divided into small-scale samples and large-scale samples according to the normalized area and scale threshold, and the solder joints smaller than or equal to the scale threshold are regarded as small-scale samples, and the solder joints larger than the scale threshold are regarded as large-scale samples; During the training of the solder joint defect detection model, the scale division is based on the normalized area of the labeled solder joints, and the denominator of the normalized area is fixed in the training configuration stage and selected from the whole image area, the candidate area corresponding to the solder joint or the device area to which the solder joint belongs; the generation rule of the scale threshold is fixed in the training configuration stage and selected from the verification set scanning rule, the generation rule according to the data distribution quantile or the generation rule according to the two-class clustering; for the solder joint samples within the preset boundary range of the scale threshold, the single attribution rule is adopted to integrate the small-scale samples, or the continuous weight distribution rule is adopted to simultaneously count the two-branch training statistics to smooth the sample attribution at the threshold boundary.

5. The machine vision-based circuit board solder joint failure identification method of claim 4, wherein, The loss function used in the training of the solder joint defect detection model is composed of a first branch loss, a second branch loss and a consistency loss. The training phase sets a first branch prediction head at the output end of the first branch to generate a prediction result of a small-scale sample and calculate the first branch loss, and sets a second branch prediction head at the output end of the second branch to generate a prediction result of a large-scale sample and calculate the second branch loss.

6. The machine vision-based circuit board solder joint failure identification method of claim 5, wherein, The consistency loss is used to constrain the consistency of the prediction results generated by the first feature extraction branch and the second feature extraction branch for the same welding spot, and the prediction results include defect category probability distribution and normalized position information; wherein the consistency of the defect category probability distribution is achieved by using symmetric divergence or cross-entropy alignment measurement after temperature smoothing of the two-branch category outputs, the consistency of the normalized position information is achieved by center point and width-height distance measurement or intersection-union regression consistency measurement; and in the training phase, confidence gating and weighted scheduling increasing with rounds are combined to suppress low-confidence alignment noise.

7. The machine vision-based circuit board solder joint failure identification method of claim 1, wherein, The first feature extraction branch includes shallow convolution for maintaining spatial resolution and feature refinement layer, and the second feature extraction branch includes down-sampling layer and context aggregation layer for expanding receptive field, and the detection head is one-stage detection head or two-stage detection head.

8. A circuit board soldering fault recognition system based on machine vision, adopting the circuit board soldering fault recognition method based on machine vision according to any one of claims 1 to 7, characterized in that, The method comprises the steps of: An image acquisition unit is configured to acquire a surface image of a circuit board under test containing exposed welding spots; A defect recognition unit is configured to load a welding defect detection model and perform inference on the surface image, and output target positions, confidence and defect categories of each welding spot; A model training unit is configured to acquire a labeled sample image set containing welding spots of different scales and train or iteratively update the welding defect detection model according to training constraints. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor executes the computer program to realize the steps of the machine vision-based circuit board welding fault identification method according to any one of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the machine vision-based circuit board welding fault identification method according to any one of claims 1-7.

Images