PCB surface defect detection method and system based on double-layer SAM model collaboration

Abstract

The invention provides a PCB surface defect detection method and system based on double-layer SAM model collaboration. The method comprises the steps that industrial image domain features and image block features of a to-be-detected PCB image are acquired; performing global semantic segmentation on the PCB image to be detected by adopting an upstream SAM model, extracting a plurality of semantic masks, and representing the semantic masks as weighted semantic mask features; on the basis of the image block features and the weighted semantic mask features, calculating the Euclidean distance between the to-be-detected PCB image and a core memory bank constructed on the basis of the normal PCB image, and obtaining a multi-level anomaly score; fusing the multi-level anomaly scores to obtain a rough anomaly score graph; a downstream SAM model is adopted, the rough anomaly score graph is used as a prompt, and an effective segmentation area mask is obtained in combination with a normal PCB structure learned by the upstream SAM model; and based on the effective segmentation region mask, pixel-level abnormal region refinement is performed on the rough abnormal score map to obtain a fine abnormal score map for category judgment and positioning of defect detection. According to the invention, high-precision defect detection in a small sample scene is realized.

Description

Technical Field

[0001] The present application relates to the technical field of printed circuit board (PCB) quality inspection, and in particular to a PCB surface defect inspection method and system based on the collaboration of a double-layer SAM model. Background Art

[0002] With the rapid development of the electronics manufacturing industry, PCB (printed circuit board) manufacturing quality, as the core basic component of electronic products, directly determines the performance and reliability of the final product. During the PCB manufacturing process, various surface defects such as scratches, stains, broken wires, short circuits, and missing components are easily generated due to factors such as unstable photolithography, uneven chemical corrosion, and machining errors. Current defect detection technologies have many shortcomings, as follows: First, the problem of sample scarcity is prominent. In industrial production scenarios, obtaining defect samples is significantly difficult and costly. Especially in the early stages of a new product line, it is often difficult to accumulate a sufficient number and variety of defect samples to support model training, resulting in limited generalization capabilities of the detection model.

[0003] Second, feature representation has limitations. Existing detection methods primarily rely on block-level feature extraction, lacking the ability to deeply analyze the semantic structure of industrial images. PCB images contain complex semantic information such as circuit layout and component distribution. Relying solely on local features cannot fully capture the correlation between defects and the surrounding environment, affecting defect identification accuracy.

[0004] Third, segmentation accuracy needs to be improved. Traditional detection methods have technical limitations in accurately locating small defects (such as micron-level scratches). Due to the small size and weak characteristic signals of such defects, accurate segmentation is difficult to achieve in complex backgrounds, and missed detections and false detections are prone to occur.

[0005] Fourth, computing efficiency is insufficient to meet demand. Some detection technologies rely on large-scale memory banks to store feature data and sample information, which results in reduced system performance in real-time detection scenarios and makes them unable to adapt to the efficient detection needs of industrial production lines.

[0006] At the application level of deep learning technology, various methods also have obvious shortcomings: CNN-based deep learning methods have a strong dependence on large-scale labeled data, and are prone to overfitting in small sample scenarios, resulting in serious degradation of detection performance; generative adversarial network methods are constrained by the problem of pattern collapse, and the coverage of generated sample types is limited, making it difficult to fully simulate various defects in PCB production; although self-supervised learning methods reduce dependence on labeled data, they have extremely high requirements for the authenticity of synthetic abnormal samples. If there are differences between the samples and the actual defects, the model's generalization ability will be limited, and it will not be able to effectively identify real production defects.

[0007] In general, existing PCB defect detection technologies still have obvious deficiencies in core technical dimensions such as small sample learning adaptability, semantic information mining depth, and small defect segmentation accuracy, and urgently need breakthroughs through technological innovation. Summary of the Invention

[0008] In view of one of the defects in the prior art, the purpose of this application is to provide a PCB surface defect detection method and system based on the collaboration of a double-layer SAM model.

[0009] In a first aspect of the present application, a PCB surface defect detection method based on a double-layer SAM model collaboration is provided, comprising: The pre-trained ResNet18 model is used to extract natural image domain features of the PCB image to be tested, and then converted into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information; Using the upstream SAM model, perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and represent them as weighted semantic mask features; Based on the image block features and the weighted semantic mask features, calculating the Euclidean distance between the PCB image to be tested and a core memory library constructed based on a normal PCB image to obtain a multi-level anomaly score; fusing the multi-level anomaly scores to obtain a rough anomaly score map; Using the downstream SAM model, the rough anomaly score map is used as a hint, and the normal PCB structure learned by the upstream SAM model is combined to obtain a valid segmentation region mask; Based on the effective segmentation area mask, pixel-level abnormal area refinement is performed on the rough abnormality score map to obtain a fine abnormality score map for category judgment and positioning of defect detection.

[0010] Optionally, the training process of the adaptive feature converter includes: Injecting Gaussian noise into the industrial image domain features of normal PCB images to generate simulated abnormal features; Using a discriminator to evaluate the difference between normal industrial image domain features and the abnormal features; Optimize the parameters of the adaptive feature converter through adversarial training ; Optionally, the upstream SAM model is used to perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and express them as weighted semantic mask features, including: After receiving the PCB image to be tested, the SAM model automatically groups the pixels by analyzing their color, texture, and shape similarity without relying on manually annotated structural labels, and obtains nine semantic masks covering nine PCB structures. Define corresponding weight values ​​according to the importance of different PCB structures; The 9 semantic masks are combined with corresponding weight values ​​to obtain weighted semantic mask features.

[0011] Optionally, the process of constructing the core memory library based on the normal PCB image includes: The image block features and weighted semantic mask features of normal PCB images are used to form a hybrid memory library; Using a greedy algorithm, find a subset such that each feature in the hybrid memory library can find a most similar feature in the subset, and the Euclidean distance between each feature in the hybrid memory library and the most similar feature is the smallest; Use this subset as the core memory bank.

[0012] Optionally, the core memory bank satisfies one or more of the following constraints: The ratio of the number of features of the subset obtained after downsampling to the total number of features of the hybrid memory library is fixed at 1%; The features in the subset cover more than 95% of the original feature space of the hybrid memory library; The features in the subset are screened using a cluster center selection strategy. Optionally, the calculating, based on the image block features and the weighted semantic mask features, the Euclidean distance between the PCB image to be tested and a core memory library constructed based on a normal PCB image to obtain a multi-level anomaly score includes: For each image block feature, calculate the Euclidean distance with all image block features in the core memory library and take the minimum distance as the initial anomaly metric; calculate the image block feature weight; and obtain the image block-level score based on the initial anomaly metric and the image block feature weight. For the weighted semantic mask feature, the Euclidean distance with all weighted semantic mask features in the core memory library is calculated, and the minimum distance is taken as the initial anomaly metric; the weight of the weighted semantic mask feature is calculated; and the mask-level score is obtained based on the initial anomaly metric and the weighted semantic mask feature.

[0013] Optionally, the fusion of the multi-level anomaly scores to obtain a rough anomaly score map includes: The image patch level score (sp) and mask level score ( ) and take the maximum value of the two as the final fusion score ( ).

[0014] Optionally, the downstream SAM model is used, with the rough anomaly score map as a hint, in combination with the normal PCB structure learned by the upstream SAM model to obtain a valid segmentation region mask, including: Select the pixel with the highest score in the rough anomaly score map as the positive prompt point; Input the rough anomaly score map as a mask prompt into the downstream SAM model; The downstream SAM model generates multiple initial segmentation regions based on the positive prompt points and the mask prompts, and in combination with the normal PCB structure semantic information learned by the upstream SAM model; Filter the initial segmented areas and remove the segmentation results where the coverage ratio exceeds a threshold β=0.2; The segmentation results retained after the above filtering together constitute the effective segmentation area mask.

[0015] Optionally, based on the effective segmentation area mask, pixel-level abnormal area refinement is performed on the rough abnormality score map to obtain a fine abnormality score map for category judgment and positioning of defect detection, including: The rough anomaly score map is superimposed with the enhancement item to obtain a fine anomaly score map; wherein the enhancement item is the enhancement coefficient, the effective segmentation area mask and the rough anomaly score Figure 3 The product of .

[0016] In a second aspect of the present application, a PCB surface defect detection system based on a double-layer SAM model collaboration is provided, comprising: Feature extraction module: Use the pre-trained ResNet18 model to extract the natural image domain features of the PCB image to be tested, convert them into industrial image domain features through an adaptive feature converter, and construct image block features containing neighborhood information; Upstream module: using the upstream SAM model to perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and represent them as weighted semantic mask features; Scoring module: Based on the image block features and weighted semantic mask features, calculate the Euclidean distance between the PCB image to be tested and the core memory library constructed based on the normal PCB image to obtain a multi-level anomaly score; Fusion module: fuses the multi-level anomaly scores to obtain a rough anomaly score map; Downstream module: adopts the downstream SAM model, takes the rough anomaly score map as a hint, and combines it with the normal PCB structure learned by the upstream SAM model to obtain the effective segmentation region mask; Refinement module: Based on the effective segmentation area mask, the rough anomaly score map is refined at the pixel level to obtain a fine anomaly score map for category judgment and positioning of defect detection.

[0017] The PCB surface defect detection method based on the collaboration of two-layer SAM models provided in this application constructs a two-layer collaborative architecture in which upstream SAM is used for semantic segmentation and downstream SAM is used for anomaly refinement. Combined with the adaptive feature converter and core memory library optimization strategy, high-precision defect detection is achieved in small sample scenarios.

[0018] Other technical effects brought about by the additional features will be further explained in the corresponding embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings: Figure 1 4 is a flow chart of a PCB surface defect detection method based on a double-layer SAM model collaboration according to an exemplary embodiment; Figure 2 1 is a structural diagram of a PCB surface defect detection system based on a double-layer SAM model collaboration according to an exemplary embodiment; Figure 3 Schematic diagram of a dual-layer SAM coordination mechanism according to an exemplary embodiment; Figure 4 1 is a structural diagram of an adaptive feature converter according to an exemplary embodiment; Figure 5 A flowchart for constructing a core memory library according to an exemplary embodiment is provided; Figure 6 FIG. 4 is a diagram showing an anomaly score fusion strategy according to an exemplary embodiment. DETAILED DESCRIPTION

[0020] The present application is described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present application, but are not intended to limit the present application in any form. It should be noted that, without departing from the concept of the present application, a number of variations and improvements may be made by those skilled in the art, and these all fall within the scope of protection of the present application. Parts not described in detail in the following examples may be implemented using existing technologies.

[0021] PCB defect detection technology still has significant deficiencies in core technical dimensions such as small sample learning adaptability, semantic information mining depth, and small defect segmentation accuracy. To address these issues, the present application provides a PCB surface defect detection method based on a collaborative dual-layer SAM model to address these issues.

[0022] Reference Figure 1 and Figure 3As shown, in one embodiment of the present application, a PCB surface defect detection method based on the collaboration of a double-layer SAM model includes: S100 uses a pre-trained ResNet18 model to extract natural image domain features of the PCB image to be tested, and converts them into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information.

[0023] S200 uses the upstream SAM model to perform global semantic segmentation on the PCB image to be tested, extracts multiple semantic masks, and represents them as weighted semantic mask features; S300 calculates the Euclidean distance between the PCB image to be tested and the core memory library built based on normal PCB images based on image block features and weighted semantic mask features to obtain a multi-level anomaly score; S400, fuses multiple levels of anomaly scores to obtain a rough anomaly score map; S500,adopts the downstream SAM model, takes the rough anomaly score map as a hint,and combines it with the normal PCB structure learned by the upstream SAM model to obtain the effective segmentation region mask; S600: Based on the effective segmentation area mask, the rough anomaly score map is refined at the pixel level to obtain a fine anomaly score map for category judgment and positioning of defect detection.

[0024] In one embodiment of the present application, a two-layer collaborative architecture is constructed in which upstream SAM is used for semantic segmentation and downstream SAM is used for anomaly refinement. Combined with the adaptive feature converter and core memory library optimization strategy, high-precision defect detection in small sample scenarios is achieved.

[0025] In some specific embodiments of the present application, S100 uses a pre-trained ResNet18 model to extract natural image domain features of the PCB image to be tested, and converts them into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information, which can be expressed as: S101 uses pre-trained ResNet18 to extract multi-level features φi,j∈R of PCB images (Cj×Hj×Wj) , where j∈[2,3]. J represents the level and i represents the pixel.

[0026] Specifically, the algorithm focuses on the middle-layer (j=2) and high-layer (j=3) features of ResNet18, retaining the local detail information of the middle-layer features (such as pad edge texture and trace width) while integrating the semantic structure information of the high-layer features (such as the "pad-trace" connection relationship). This achieves multi-level feature coverage from pixel-level details to structural-level semantics, providing more comprehensive feature support for subsequent defect detection.

[0027] Specifically, the lightweight architecture and targeted feature conversion based on ResNet18 can control the computational complexity while ensuring the feature expression capability, avoid the efficiency loss caused by deep networks, and lay an efficient feature foundation for subsequent memory library construction and real-time detection.

[0028] S102 converts natural image domain features into industrial image domain features through the adaptive feature converter Fθ .

[0029] Specifically, the natural image features of the pre-trained model are domain-converted through an adaptive feature converter, which effectively alleviates the "distribution offset problem" of the pre-trained model on industrial PCB images (for example, the low texture and high contrast characteristics of PCB images are significantly different from natural images), making the extracted features more consistent with the visual attributes of industrial structures such as pads and traces, and improving the targeted expression ability of the features for PCB scenes.

[0030] S103 Constructing image block feature representation containing neighborhood information .

[0031] Specifically, by constructing image block features containing neighborhood information and associating single image block features with surrounding area features, the spatial propagation characteristics of defects (such as the continuous distribution of scratches and the regional correlation of short-circuit defects) can be effectively captured, avoiding the "local misjudgment" caused by isolated analysis of single block features, and improving the feature characterization capabilities of structural defects (such as broken wires and short circuits).

[0032] The above-mentioned embodiments of the present application achieve high-precision, high-adaptability, and high-efficiency PCB defect detection support capabilities through S101 comprehensive feature extraction, S102 domain adaptive optimization, and S103 spatial correlation modeling. This not only improves the feature characterization accuracy of various defects, but also ensures the real-time and robustness required for industrial scenarios, providing a more reliable feature foundation for subsequent defect detection tasks (such as classification and positioning).

[0033] In a specific embodiment, the conversion process of S101-S103 in the feature domain can be implemented using the following computer programming language: class AdaptiveFeatureTransformer(nn.Module): def __init__(self, input_dim, hidden_dim=256): super().__init__() self.encoder = nn.Sequential( nn.Conv2d(input_dim, hidden_dim, 3, padding= 1), nn.ReLU(), nn.Conv2d(hidden_dim, input_dim, 3, padding= 1) ) def forward(self, x): return x + self.encoder(x) # residual connection In some specific implementations of this application, the above-mentioned pre-trained ResNet18, such as Figure 4 As shown, the following training method is adopted: First, the industrial image domain features after the normal PCB image features are connected through Conv2D-ReLU and Conv2D residual connections Inject Gaussian noise to generate simulated anomaly features: . Among them, ε~N(0,σ2) represents Gaussian noise, φi- represents abnormal features, Indicates a normal PCB image; Specifically, traditional anomaly detection often relies on a large number of real abnormal samples, but the number of abnormal samples in PCB production (such as short circuits, open circuits, stains, etc.) is often very small and the types are varied, making them difficult to collect. This embodiment directly uses the normal sample features of normal PCB images ( ) injects Gaussian noise (ε~N(0,σ²)) to generate simulated abnormal features ( ), it is possible to construct "normal-abnormal" comparison sample pairs without the need for actual abnormal data. This approach cleverly circumvents the pain point of a shortage of abnormal samples, especially avoiding the cost of manually labeling abnormal samples.

[0034] Then, the discriminator Du is used to evaluate the normal sample characteristics and the difference in abnormal characteristics φi-; Finally, the converter parameters are optimized through adversarial training Specifically, through the adversarial game between the discriminator Du and the converter (the objective function of minθmaxψ), a closed loop of generating simulated anomalies, discriminator differentiation, and converter optimization feature expression is achieved: the discriminator Du needs to distinguish normal features as much as possible ( ) and simulated abnormal characteristics ( ), promoting its own sensitivity to the difference between normal and abnormal; the converter parameter θ indirectly optimizes the expression of the feature space by minimizing the discriminator's ability to distinguish (that is, making simulated abnormalities more difficult to identify) - making normal features purer and the difference between abnormal features (even if simulated) and normal features more significant.

[0035] In a specific embodiment, the above training of ResNet18 can be implemented using the following computer programming languages: # Use pre-trained ResNet18 backbone = ResNet18(pretrained=True) features = backbone.extract_features(input_image, layers= [2, 3]).

[0036] It is worth noting that in the above training process, noise injection and adversarial training are performed directly in the feature space (rather than the original image), which is more in line with the working logic of the deep learning model: the model ultimately relies on features rather than raw pixels for judgment, and strengthens the differences between normal and abnormal at the feature level, which can more efficiently improve the model's ability to perceive abnormalities.

[0037] The above-mentioned embodiments of the present application, under the constraint of scarcity of abnormal samples, actively build the distinction between normal and abnormal through adversarial generation at the feature level, which not only solves the data problem, but also specifically strengthens the model's perception of abnormal features. It is particularly suitable for the high-precision detection needs of tiny and unknown anomalies in industrial quality inspection scenarios such as PCBs.

[0038] In order to make the model focus on key structures more accurately and adapt to the actual needs of industrial detection. In some specific implementations of this application, S200 uses the upstream SAM model to perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and express them as weighted semantic mask features, such as Figure 3 As shown, the following steps can be taken: S201, after receiving the PCB image to be tested, the SAM model automatically groups the pixels by analyzing the color, texture, and shape similarity in the image without relying on manually labeled structural labels. Semantic masks, including 9 PCB structures.

[0039] Exemplarily, the nine types of PCB structures include solder joints, conductive lines, chip pins, pads, vias, solder resist layers, silk screen layers, edge blank areas, non-functional markings, and the like.

[0040] S202, defining corresponding weight values ​​according to the importance of different PCB structures, where the weights reflect the importance of different regions (structures).

[0041] Specifically, the weight value here can be set based on experience.

[0042] S203 , performing weighted calculation on the 9 semantic masks and the corresponding weight values ​​to obtain a weighted semantic mask feature.

[0043] Specifically, anomalies (such as short circuits, cold solder joints, and fractures) in key structures (such as solder joints, conductive lines, and chip pins) may directly lead to PCB functional failure and are the core focus of inspection. Anomalies in secondary structures (such as blank areas on the edges, non-functional markings, and slightly stained areas) have less impact on PCB performance and may even be irrelevant interference (such as dust and reflections from photography). If the semantic mask features are not weighted, the model will regard the features of all structures as equally important, which may result in: the abnormal features of the key structures being "diluted" by the normal features of the secondary structures, reducing detection sensitivity; and slight interference in the secondary structures being misjudged as anomalies, increasing the false detection rate. The essence of weighting is to strengthen the contribution of key structural features and weaken the interference of secondary structures through manually or automatically defined weights, so that the model can focus more on "areas that really need attention" in subsequent inspections.

[0044] In the above embodiment of the present application, steps S201-S203 adopt a combination of "SAM automatic segmentation + weighting", which lowers the threshold for industrial implementation. Specifically, this method does not require manual labeling of structural labels and can adapt to scenarios with changing PCB models, thereby improving the flexibility and efficiency of the detection process. At the same time, this combination method also improves the accuracy and robustness of detection. Through weighted operations, abnormal signals of key structures are strengthened and secondary interference is suppressed, so that the model can more accurately identify "anomalies that truly affect PCB functions", reducing the occurrence of missed detections and false detections.

[0045] In a specific embodiment of the present application, the upstream SAM semantic segmentation process can be implemented using the following computer programming language: # Use SAM for global segmentation sam_predictor = SAM(checkpoint_path="sam_vit_h.pth") masks = sam_predictor.predict_masks(image, n_masks=9) # Calculate mask weight weights = compute_mask_weights(masks, features) weighted_features =features weights In order to keep the feature distribution in the core memory library representative and diverse, in some specific implementations of this application, a core memory library is constructed, such as Figure 5 As shown, the following steps can be taken: First, build a hybrid memory library , where Mpatch is the image block level feature, It is a mask-level feature.

[0046] Specifically, Mpatch (image block-level features) segments the PCB image into multiple local patches and extracts features (such as texture, color, and local structural features) from each patch. These features are fine-grained and can reflect local image details, such as slight deformation of solder joints and localized fractures in circuits.

[0047] Mmask (mask-level features): These features are extracted based on the semantic masks mentioned above (e.g., the masks for the nine PCB structures). They represent the overall characteristics of the entire structural area, such as the overall outline of the chip area or the global distribution of the circuit network. These features are coarse-grained and focus on structural information.

[0048] A single feature cannot capture both local details and global structure. For example, using only block features may overlook inter-structural correlations, while using only mask features may miss details of minor anomalies. By integrating these two features, M captures subtle local anomaly details while preserving global structural information, providing a more comprehensive feature foundation for subsequent anomaly detection.

[0049] Then, the greedy algorithm is used to optimize the target To achieve downsampling. Specifically, the formula means: for each feature m in M, find the feature n closest to it in the candidate subset Mc (the distance is expressed in L2 norm). Measure); calculate the maximum of these "closest distances" (i.e. ), which reflects the "maximum coverage error" of Mc to M - that is, the distance between the feature farthest from Mc in the original feature and the nearest element in Mc; find the subset Mc (that is, argmin) that can minimize this "maximum coverage error", that is, let Mc "cover" the distribution of the original feature M as evenly as possible, and at the same time Mc is a subset of M.

[0050] Therefore, the subset finally obtained by the greedy algorithm enables each feature in the mixed memory library to find a most similar feature in the subset, and the Euclidean distance between each feature in the mixed memory library and the most similar feature is the smallest.

[0051] Of course, in order to make the downsampled core memory library more representative and diverse, in some specific embodiments, the following constraints are imposed: The downsampling ratio is fixed at 1% to ensure computational efficiency; Ensure that the compressed features cover more than 95% of the original feature space; A cluster center selection strategy is adopted to prioritize the retention of representative features.

[0052] In the above-mentioned embodiment of the present application, the combined effect of the three constraints can not only significantly reduce the computational load of the core memory library through a 1% downsampling ratio, thereby ensuring the real-time efficiency of industrial detection; but also can prioritize the retention of representative features by covering more than 95% of the original feature space in conjunction with the clustering center selection strategy, thereby effectively avoiding the loss of key features. In this way, while controlling the computational cost, it ensures that the feature distribution of the core memory library is both broadly representative and retains sufficient diversity, thereby providing reliable support for the accuracy and robustness of subsequent defect detection.

[0053] In a specific embodiment, the construction of the core memory library can be implemented using the following computer programming languages: # Build a hybrid memory library patch_features=extract_patch_features(normal_images) mask_features=extract_mask_features(normal_images,sam_masks) hybrid_memory=torch.cat([patch_features,mask_features],dim=0) Among them, the greedy downsampling algorithm used can be implemented in the following computer programming languages: def greedy_subsampling(memory_bank, subsample_ratio=0.01): n_samples = int(len(memory_bank) subsample_ratio) selected_indices = [0] # Initially select the first sample for _ in range(n_samples - 1): distances = [ ] for i in range(len(memory_bank)): if i in selected_indices: continue # Calculate the minimum distance to the selected sample min_dist = min([torch.norm(memory_bank[i] - memory_bank[j]) forj in selected_indices]) distances.append((min_dist, i)) # Select the sample with the largest distance best_idx = max(distances) selected_indices.append(best_idx) return memory_bank[selected_indices] Based on the core memory library set in the above embodiment, in some specific implementations of the present application, S300, based on the image block features and weighted semantic mask features, calculates the Euclidean distance between the PCB image to be tested and the core memory library constructed based on the normal PCB image to obtain a multi-level anomaly score, such as Figure 6 As shown, the following steps can be taken: S301, calculate the Euclidean distance between the image block feature of the PCB image to be tested and all the image feature blocks in the core memory library; take the minimum distance as the initial abnormality measurement of the image block. Use the adaptive weight mechanism to calculate the weight: , is the weight, m ​​is the image block feature of the core memory bank, m test is the image block feature of the PCB image to be tested, is the initial anomaly measure, Indicates the core memory bank The weights are fused with the initial anomaly metric to obtain the image patch-level score sp.

[0054] S302: Similarly, the same calculation steps as 301 are used to obtain the initial anomaly measurement and weight, and the mask-level score is calculated. .

[0055] In some specific embodiments of the present application, in step S400, the multi-level anomaly scores are fused to obtain a rough anomaly score map. The following steps can be used: fusion of the image block level score sp and the mask level score .

[0056] Specifically, the image block-level score sp is used to score the degree of abnormality of the image block. It focuses on local details and can capture small local anomalies, such as a single solder joint failure or a slight break in a certain circuit, but may be affected by local noise, such as isolated pixel anomalies. Mask-level scoring : Anomaly score based on semantic masks (e.g., overall structure of "chip region" or "circuit network"). It reflects global structural anomalies (e.g., overall morphological anomalies of a certain type of structure, missing key structures), but is less sensitive to local minor anomalies.

[0057] The essence of the “maximum value” fusion method is “not to miss any possible abnormal signal”: when the image block level score sp is high (there is obvious abnormality locally), even if the mask level score Low (the global structure appears normal), final score sp will also be taken to ensure that local minor anomalies are not ignored; when the mask level score High (there are significant anomalies in the global structure), even if the local block-level score sp is low (no obvious anomalies are found in the local details), the final score Will take ,to avoid missing global structural anomalies (such as overall chip offset).

[0058] PCB anomalies can be localized minor defects (such as pinholes or small areas of corrosion) or global structural issues (such as overall circuit misalignment or missing components). The max function in the above embodiment of this application ensures that anomalies of both scales are captured, avoiding the one-sidedness of scoring based on a single scale.

[0059] For the case of "local anomalies but the global structure appears normal" (such as a small break in a certain line that does not affect the overall structural morphology), the high value of sp will dominate the final score to prevent missed detection; for the case of "global anomalies but local details do not trigger a high score" (such as the overall chip offset, but a single pixel block does not exceed the anomaly threshold), A high value of will dominate the score, also to avoid missed detections.

[0060] The "maximum value" operation is simple and efficient, without the need for complex weight calculations. It can quickly output fusion results while ensuring coverage of various types of anomalies, meeting the "high sensitivity + high efficiency" requirements in industrial detection.

[0061] Early fusion score Although it takes into account the information of local blocks and global masks, there are two potential limitations: Boundary blur problem: the image block-level score sp may give a blurry score (neither high nor low) at the edge of the abnormal area, while the mask-level score Focusing on the overall structure, it is difficult to distinguish boundary details; risk of semantic misjudgment: the features of certain local areas may be similar to anomalies (such as local brightness anomalies caused by reflections in the image), but from a global semantic perspective, they are not truly abnormal (such as reflections in non-critical areas), and the early fusion score may be misjudged as anomalies. Therefore, in some specific embodiments of this application, S500 uses the downstream SAM model, uses the rough anomaly score map as a prompt, and combines it with the normal PCB structure learned by the upstream SAM model to obtain an effective segmentation area mask. The following steps can be used: S501, selecting the pixel with the highest score in the rough anomaly score map as a positive prompt point.

[0062] Specifically, the model's attention is directly directed to the core area that is "most likely to be anomaly", avoiding indiscriminate search of the entire image by the downstream SAM, reducing invalid calculations, and improving segmentation efficiency.

[0063] S502: Input the rough anomaly score map as a mask prompt into the downstream SAM model.

[0064] Specifically, it is equivalent to giving the downstream SAM a "preliminary range guide", allowing it to perform more detailed segmentation within the range rather than exploring from scratch, greatly improving the targetedness and accuracy of the segmentation.

[0065] S503, the downstream SAM model generates multiple initial segmentation regions based on the positive cue points and mask cues, and combined with the normal PCB structure semantic information learned by the upstream SAM model.

[0066] S504 , filtering the initial segmented areas, and eliminating the segmentation results where the coverage area ratio exceeds a threshold β=0.2.

[0067] Specifically, through threshold constraints, large areas that do not meet the "local anomaly" characteristics are filtered out, reducing false positive annotations (such as misjudging slight brightness differences in the entire image as anomalies), ensuring that the retained segmentation results are true "small-scale, high-risk" anomalies.

[0068] S505: The segmentation results retained after the above filtering together constitute a valid segmentation area mask.

[0069] It's worth noting that the upstream SAM model has already learned the structural semantics of a normal PCB through previous steps, such as the normal routing of traces, the typical shape of solder joints, and the relative positional relationships of various structures. Therefore, the upstream SAM model provides a baseline for normality. During segmentation, the downstream SAM compares the "current region features" with the "normal structural features learned upstream" to determine whether the region deviates from normal morphology, thus avoiding misclassifying "normal but morphologically unusual structures" as abnormal. Of course, the upstream SAM model also constrains the rationality of segmentation: PCBs are highly structured products, and the shape, size, and position of each component follow a fixed logic (e.g., uniform trace width and tight solder joint-to-circuit connections). Leveraging these normal semantics, the downstream SAM model can correct "false anomalies" in the roughness score map, such as local deformation caused by the camera angle, for actual normal structures, ensuring that the segmentation results are more consistent with the PCB's physical structure.

[0070] In order to more accurately support the classification and location of PCB defects, some specific embodiments of this application, S600, based on the effective segmentation area mask, perform pixel-level abnormal area refinement on the rough abnormality score map to obtain a fine abnormality score map for the classification and location of defect detection, can adopt the following steps: The rough anomaly score map is superimposed with the enhancement item to obtain the fine anomaly score map; the enhancement item is the enhancement coefficient, the effective segmentation area mask and the rough anomaly score. Figure 3 The product of them. That is: , where α=0.3 is the enhancement coefficient.

[0071] The above-described embodiments of this application can precisely amplify valid signals and suppress invalid interference. By incorporating prior information about valid segmented areas into the scoring calculation, the resulting refined anomaly score map maintains pixel-level positioning accuracy while enhancing the distinction between defective and normal areas. This provides a high-quality feature foundation for the precise location and classification of printed circuit board (PCB) defects, perfectly meeting the core requirements of "high precision and low false positives" in industrial quality inspection.

[0072] In a specific embodiment, the detailed implementation process of the testing phase can be implemented using the following computer programming languages: Multi-level anomaly score calculation: Image patch level scoring: Python def compute_patch_scores(test_features, core_memory): patch_scores = [ ] for patch_feat in test_features: distances = torch.norm(patch_feat - core_memory, dim= 1) min_distance= torch.min(distances) patch_scores.append(min_distance) return torch.stack(patch_scores) Mask-level scoring: def compute_mask_scores(test_image, sam_masks, core_memory): mask_scores = [ ] for mask in sam_masks: mask_feat = extract_mask_feature(test_image, mask) distances = torch.norm(mask_feat - core_memory, dim= 1) min_distance= torch.min(distances) mask_scores.append(min_distance) return torch.stack(mask_scores) Rating Fusion: def fuse_scores(patch_scores, mask_scores): # Upsample to the same size patch_map = F.interpolate(patch_scores, size=(H, W)) mask_map = F.interpolate(mask_scores, size=(H, W)) # Take the maximum value fusion fused_map = torch.max(patch_map, mask_map) return fused_map Downstream SAM refinement def downstream_sam_refinement(test_image, coarse_map, alpha=0.3): # Select high score points as hints high_score_points = select_high_score_points(coarse_map, top_k=5) SAM Refinement refined_mask = sam_predictor.predict( image=test_image, point_prompts=high_score_points, mask_prompt =coarse_map > threshold ) # Result fusion final_map = coarse_map + alpha refined_mask coarse_map returnfinal_map Based on the same technical concept, in other embodiments of the present application, a PCB surface defect detection system 100 based on the collaboration of a double-layer SAM model is provided. Figure 2 Shown, including: Feature extraction module 110: uses a pre-trained ResNet18 model to extract natural image domain features of the PCB image to be tested, converts them into industrial image domain features through an adaptive feature converter, and constructs image block features containing neighborhood information; Upstream module 120: using the upstream SAM model to perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and represent them as weighted semantic mask features; Scoring module 130: Based on the image block features and weighted semantic mask features, calculates the Euclidean distance between the PCB image to be tested and the core memory library constructed based on the normal PCB image to obtain a multi-level anomaly score; Fusion module 140: Fusion of multiple levels of anomaly scores to obtain a rough anomaly score map; Downstream module 150: adopts the downstream SAM model, takes the rough anomaly score map as a hint, and combines it with the normal PCB structure learned by the upstream SAM model to obtain a valid segmentation region mask; Refinement module 160: Based on the effective segmentation area mask, the rough anomaly score map is refined at the pixel level to obtain a fine anomaly score map for category judgment and positioning of defect detection.

[0073] The specific implementation technology of each module / unit in the above example of this application can refer to the corresponding steps of the PCB surface defect detection method based on the collaboration of the double-layer SAM model in the above embodiment, and will not be repeated here.

[0074] The following further illustrates the present application in conjunction with specific application examples / comparative examples in order to better understand the above technical solutions of the present application. It should be understood that the following are merely partial examples and are not intended to limit the present application.

[0075] The key parameter configuration and optimization of the PCB surface defect detection method based on the collaboration of the double-layer SAM model in the above embodiment are shown in Table 1.

[0076]

[0077] Application scenario 1: PCB trace defect detection. Optimize parameters for trace defects by increasing the number of masks to 12 to capture thin and long traces, using feature layers 1, 2, and 3, and setting the enhancement factor to 0.5.

[0078] Application scenario 2 is component missing detection. Optimize parameters for component missing by increasing the number of masks to 12 to capture the component area, using a larger patchsize of 3.2 and a slightly larger memory bank 'memory_ratio' of 0.02.

[0079] The above applications verify the feasibility of this application method.

[0080] To verify the technical effectiveness of the method in this embodiment, a comparative experiment was conducted. The experiment was based on the MVTecAD standard dataset, a self-built dataset of 10 defective PCBs, and a small sample test set (1-shot and 5-shot). The results were verified by focusing on indicators such as classification AUROC and segmentation AUROC. The comparison results are shown in Table 2:

[0081] The comparison results in Table 2 show that the classification and segmentation AUROC of the proposed method are significantly ahead, and its generalization in small sample scenarios (1-shot reaches 88.3%, 5-shot reaches 92.1%) and detection speed (22.7FPS) are obvious advantages.

[0082] In order to verify the accuracy gain of each module component, an ablation experiment was carried out, and the results are shown in Table 3.

[0083]

[0084] It can be seen that the ablation experiment verifies that each component is useful and can generate gains.

[0085] In the description of the embodiments of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0086] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or to implicitly indicate the quantity of the technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0087] In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly specified and specifically defined. In the present application, unless otherwise clearly specified and defined, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0088] In the embodiments of the present application, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0089] The above describes some specific embodiments of the present application. It should be understood that the present application is not limited to the specific embodiments described above, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the substantive content of the present application. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. A PCB surface defect detection method based on a double-layer SAM model collaboration is characterized by: include: The pre-trained ResNet18 model is used to extract natural image domain features of the PCB image to be tested, and then converted into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information; Using the upstream SAM model, perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and represent them as weighted semantic mask features; Based on the image block features and the weighted semantic mask features, calculating the Euclidean distance between the PCB image to be tested and a core memory library constructed based on a normal PCB image to obtain a multi-level anomaly score; fusing the multi-level anomaly scores to obtain a rough anomaly score map; Using the downstream SAM model, the rough anomaly score map is used as a hint, and the normal PCB structure learned by the upstream SAM model is combined to obtain a valid segmentation region mask; Based on the effective segmentation area mask, pixel-level abnormal area refinement is performed on the rough abnormality score map to obtain a fine abnormality score map for category judgment and positioning of defect detection.

2. The PCB surface defect detection method based on the double-layer SAM model collaboration according to claim 1 is characterized in that: The training process of the adaptive feature converter includes: Injecting Gaussian noise into the industrial image domain features of normal PCB images to generate simulated abnormal features; Using a discriminator to evaluate the difference between normal industrial image domain features and the abnormal features; The parameters of the adaptive feature converter are optimized through adversarial training.

3. The PCB surface defect detection method based on the double-layer SAM model collaboration according to claim 1 is characterized in that: The upstream SAM model is used to perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and express them as weighted semantic mask features, including: After receiving the PCB image to be tested, the SAM model automatically groups the pixels by analyzing their color, texture, and shape similarity without relying on manually annotated structural labels, and obtains nine semantic masks covering nine PCB structures. Define corresponding weight values ​​according to the importance of different PCB structures; The 9 semantic masks are multiplied by corresponding weight values ​​to obtain weighted semantic mask features.

4. The PCB surface defect detection method based on the double-layer SAM model collaboration according to claim 1 is characterized in that: The process of building the core memory library based on a normal PCB image includes: The image block features and weighted semantic mask features of normal PCB images are used to form a hybrid memory library; Using a greedy algorithm, find a subset such that each feature in the hybrid memory library can find a most similar feature in the subset, and the Euclidean distance between each feature in the hybrid memory library and the most similar feature is the smallest; Use this subset as the core memory bank.

5. The PCB surface defect detection method based on the collaboration of the double-layer SAM model according to claim 4 is characterized in that: The core memory library satisfies one or more of the following constraints: The ratio of the number of features of the subset obtained after downsampling to the total number of features of the hybrid memory library is fixed at 1%; The features in the subset cover more than 95% of the original feature space of the hybrid memory library; The features in the subset are screened using a cluster center selection strategy.

6. The PCB surface defect detection method based on double-layer SAM model collaboration according to claim 1 is characterized in that: The step of calculating the Euclidean distance between the PCB image to be tested and a core memory library constructed based on a normal PCB image based on the image block features and the weighted semantic mask features to obtain a multi-level anomaly score includes: For each image block feature, calculate the Euclidean distance with all image block features in the core memory library and take the minimum distance as the initial anomaly metric; calculate the image block feature weight; and obtain the image block-level score based on the initial anomaly metric and the image block feature weight. For the weighted semantic mask feature, the Euclidean distance with all weighted semantic mask features in the core memory library is calculated, and the minimum distance is taken as the initial anomaly metric; the weight of the weighted semantic mask feature is calculated; and the mask-level score is obtained based on the initial anomaly metric and the weighted semantic mask feature.

7. The PCB surface defect detection method based on double-layer SAM model collaboration according to claim 1 is characterized in that: The fusion of the multi-level anomaly scores to obtain a rough anomaly score map includes: The image patch-level score and the mask-level score are compared, and the maximum of the two is taken as the final fusion score.

8. The PCB surface defect detection method based on double-layer SAM model collaboration according to claim 1 is characterized in that: The downstream SAM model is used, with the rough anomaly score map as a hint, combined with the normal PCB structure learned by the upstream SAM model to obtain an effective segmentation region mask, including: Select the pixel with the highest score in the rough anomaly score map as the positive prompt point; Input the rough anomaly score map as a mask prompt into the downstream SAM model; The downstream SAM model generates a plurality of initial segmentation regions based on the positive prompt points and the mask prompts and in combination with the normal PCB structure semantic information learned by the upstream SAM model; Filtering the initial segmented areas to remove segmentation results where the coverage area ratio exceeds a threshold; The segmentation results retained after the above filtering together constitute the effective segmentation area mask.

9. The PCB surface defect detection method based on double-layer SAM model collaboration according to claim 1 is characterized in that: Based on the effective segmentation area mask, pixel-level abnormal area refinement is performed on the rough abnormality score map to obtain a fine abnormality score map for category judgment and positioning of defect detection, including: The rough anomaly score map is superimposed with the enhancement item to obtain a fine anomaly score map; wherein the enhancement item is the product of the enhancement coefficient, the effective segmentation area mask and the rough anomaly score map.

10. The PCB surface defect detection system based on the collaboration of the double-layer SAM model is characterized by: include: Feature extraction module: Use the pre-trained ResNet18 model to extract the natural image domain features of the PCB image to be tested, convert them into industrial image domain features through an adaptive feature converter, and construct image block features containing neighborhood information; Upstream module: using the upstream SAM model to perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and represent them as weighted semantic mask features; Scoring module: Based on the image block features and weighted semantic mask features, calculate the Euclidean distance between the PCB image to be tested and the core memory library constructed based on the normal PCB image to obtain a multi-level anomaly score; Fusion module: fuses the multi-level anomaly scores to obtain a rough anomaly score map; Downstream module: adopts the downstream SAM model, takes the rough anomaly score map as a hint, and combines it with the normal PCB structure learned by the upstream SAM model to obtain the effective segmentation region mask; Refinement module: Based on the effective segmentation area mask, the rough anomaly score map is refined at the pixel level to obtain a fine anomaly score map for category judgment and positioning of defect detection.

Images