PCB surface defect detection method and system based on double-layer SAM model cooperation

Abstract

The application provides a PCB surface defect detection method and system based on a double-layer SAM model collaboration. The method comprises: obtaining industrial image domain features and image block features of a to-be-tested PCB image; using an upstream SAM model to perform global semantic segmentation on the to-be-tested PCB image, extract multiple semantic masks, and represent them as weighted semantic mask features; based on the image block features and the weighted semantic mask features, calculating the Euclidean distance between the to-be-tested PCB image and a core memory library constructed based on normal PCB images, and obtaining multi-level anomaly scores; fusing the multi-level anomaly scores to obtain a coarse anomaly score map; using a downstream SAM model, taking the coarse anomaly score map as a prompt, and combining the normal PCB structure learned by the upstream SAM model to obtain an effective segmentation region mask; based on the effective segmentation region mask, performing pixel-level anomaly region refinement on the coarse anomaly score map to obtain a fine anomaly score map, which is used for category judgment and positioning of defect detection. The application realizes high-precision defect detection in a small sample scenario.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of printed circuit board (PCB) quality detection, in particular to a PCB surface defect detection method and system based on double-layer SAM model cooperation. BACKGROUND

[0002] With the rapid development of electronic manufacturing industry, PCB (printed circuit board) as the core basic component of electronic products, its manufacturing quality directly determines the performance and reliability of the final product. In the PCB manufacturing process, due to the instability of photoetching process, uneven chemical corrosion, mechanical processing error and other factors, various surface defects such as scratches, stains, broken lines, short circuits and missing components are prone to occur. The current existing defect detection technology has many shortcomings, as follows: first, the sample scarcity problem is prominent. In the industrial production scene, it is difficult and costly to obtain defect samples, especially in the early stage of new product line, it is often difficult to accumulate enough number and type of defect samples to support model training, which limits the generalization ability of the detection model.

[0003] Second, the feature representation has limitations. The existing detection method mainly extracts image block level features, and lacks deep analysis ability of the semantic structure of industrial images. PCB images contain complex circuit layout, component distribution and other semantic information, and only relying on local features cannot fully capture the relevance of defects and the surrounding environment, affecting the defect recognition accuracy.

[0004] Third, the segmentation accuracy needs to be improved. Traditional detection methods have technical limitations in precise positioning of small defects (such as micron-level scratches). Due to the small size and weak feature signal of such defects, it is difficult to achieve accurate segmentation in complex background, and it is easy to miss detection and false detection.

[0005] Fourth, the computing efficiency is difficult to meet the demand. Some detection technologies rely on large-scale memory to store feature data and sample information, which leads to a decline in system performance in real-time detection scenarios, and cannot adapt to the high-efficiency detection needs of industrial production lines.

[0006] In the application level of deep learning technology, various methods also have obvious shortcomings: the deep learning method based on CNN has strong dependence on large-scale labeled data, and in the small sample scene, overfitting is easy to occur, which leads to serious degradation of detection performance; the generative adversarial network method is restricted by the mode collapse problem, and the coverage of generated sample types is limited, which is difficult to fully simulate various defects in PCB production; although the self-supervised learning method reduces the dependence on labeled data, the authenticity of synthetic abnormal samples is extremely high, and if the sample and the actual defect are different, the generalization ability of the model will be limited, and the real production defects cannot be effectively identified.

[0007] Overall, the existing PCB defect detection technology still has obvious deficiencies in small sample learning adaptability, semantic information mining depth, and small defect segmentation precision in core technical dimensions, and breakthroughs are urgently needed through technical innovation. SUMMARY

[0008] For the defects / one of the prior art, the purpose of the present application is to provide a PCB surface defect detection method and system based on double-layer SAM model cooperation.

[0009] The first aspect of the present application provides a PCB surface defect detection method based on double-layer SAM model cooperation, comprising:

[0010] Using a pre-trained ResNet18 model to extract natural image domain features of the PCB image to be tested, and converting them into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information;

[0011] Using an upstream SAM model to perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and represent them as weighted semantic mask features;

[0012] Based on the image block features and weighted semantic mask features, calculate the Euclidean distance between the PCB image to be tested and the core memory library constructed based on normal PCB images, and obtain multi-level anomaly scores;

[0013] Fusing the multi-level anomaly scores, a rough anomaly score map is obtained;

[0014] Using a downstream SAM model, taking the rough anomaly score map as a prompt, combining the normal PCB structure learned by the upstream SAM model, obtaining an effective segmentation region mask;

[0015] Based on the effective segmentation region mask, performing pixel-level anomaly region refinement on the rough anomaly score map to obtain a fine anomaly score map for class judgment and positioning of defect detection.

[0016] Optionally, the training process of the adaptive feature converter includes:

[0017] Injecting Gaussian noise into the industrial image domain features of normal PCB images to generate simulated abnormal features;

[0018] Using a discriminator to evaluate the difference between normal industrial image domain features and the abnormal features;

[0019] Optimizing the parameters of the adaptive feature converter through adversarial training ;

[0020] Optionally, the upstream SAM model is used to perform global semantic segmentation on the to-be-tested PCB image, extract a plurality of semantic masks, and represent as weighted semantic mask features, comprising:

[0021] After the SAM model receives the to-be-tested PCB image, without relying on manually labeled structure labels, the SAM model automatically groups the pixels in the image according to color, texture, and shape similarity to obtain nine semantic masks, which contain nine types of PCB structures;

[0022] According to the importance of different PCB structures, corresponding weight values are defined;

[0023] The nine semantic masks and the corresponding weight values are used to obtain weighted semantic mask features.

[0024] Optionally, the construction process of the core memory library constructed based on normal PCB images comprises:

[0025] The image block features and the weighted semantic mask features of the normal PCB images are used to form a hybrid memory library;

[0026] A greedy algorithm is used to find a subset such that each feature in the hybrid memory library can find a most similar feature in the subset, and the Euclidean distance between each feature in the hybrid memory library and the most similar feature is minimized;

[0027] The subset is used as the core memory library.

[0028] Optionally, the core memory library satisfies one or more of the following constraint conditions:

[0029] The proportion of the number of features of the subset obtained after downsampling to the total number of features of the hybrid memory library is fixed at 1%;

[0030] The features in the subset cover more than 95% of the original feature space of the hybrid memory library;

[0031] The features in the subset are selected using a cluster center selection strategy,

[0032] Optionally, based on the image block features and the weighted semantic mask features, the Euclidean distance between the to-be-tested PCB image and the core memory library constructed based on normal PCB images is calculated to obtain a multi-level anomaly score, comprising:

[0033] For the image block features, the Euclidean distance with all image block features of the core memory library is calculated, and the minimum distance is taken as the initial anomaly measure; the image block feature weight is calculated; and the image block level score is obtained based on the initial anomaly measure and the image block feature weight;

[0034] For weighted semantic mask features, calculate the Euclidean distance with all weighted semantic mask features in the core memory library, and take the minimum distance as the initial anomaly metric; calculate the weights of the weighted semantic mask features; and obtain a mask-level score based on the initial anomaly metric and the weighted semantic mask features.

[0035] Optionally, the fusion of the multi-level anomaly scores to obtain a coarse anomaly score map includes:

[0036] Image block-level scoring (sp) and mask-level scoring ( The two scores are compared, and the maximum value is taken as the final fusion score. ).

[0037] Optionally, the downstream SAM model, using the coarse anomaly score map as a cue, combined with the normal PCB structure learned by the upstream SAM model, obtains an effective segmentation region mask, including:

[0038] Select the pixel with the highest score in the roughness anomaly rating map as the positive prompt point;

[0039] The rough anomaly score map is used as a mask prompt input into the downstream SAM model;

[0040] The downstream SAM model generates multiple initial segmentation regions based on the positive cue points and the mask cue, and in combination with the normal PCB structure semantic information learned by the upstream SAM model.

[0041] The initial segmented region is filtered out, and segmentation results whose coverage area ratio exceeds the threshold β=0.2 are removed;

[0042] The segmentation results retained after the above filtering together constitute the effective segmentation region mask.

[0043] Optionally, based on the effective segmentation region mask, pixel-level anomaly region refinement is performed on the coarse anomaly scoring map to obtain a fine anomaly scoring map, which is used for defect detection category judgment and location, including:

[0044] The coarse anomaly score map is overlaid with the enhancement terms to obtain the fine anomaly score map; wherein, the enhancement terms are the enhancement coefficient, the effective segmentation region mask, and the coarse anomaly score. Figure Three The product of the products.

[0045] A second aspect of this application provides a PCB surface defect detection system based on a two-layer SAM model collaboration, comprising:

[0046] The feature extraction module extracts natural image domain features of the to-be-tested PCB image using a pre-trained ResNet18 model, and converts the features into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information.

[0047] The upstream module adopts an upstream SAM model to perform global semantic segmentation on the to-be-tested PCB image, extracts a plurality of semantic masks, and represents the masks as weighted semantic mask features.

[0048] The scoring module calculates the Euclidean distance between the to-be-tested PCB image and a core memory library constructed based on normal PCB images based on the image block features and the weighted semantic mask features, and obtains multi-level abnormality scores.

[0049] The fusion module fuses the multi-level abnormality scores to obtain a coarse abnormality score map.

[0050] The downstream module adopts a downstream SAM model to obtain an effective segmentation region mask by taking the coarse abnormality score map as a prompt and combining the normal PCB structure learned by the upstream SAM model.

[0051] The refinement module performs pixel-level abnormal region refinement on the coarse abnormality score map based on the effective segmentation region mask to obtain a fine abnormality score map for class judgment and positioning of defect detection.

[0052] The PCB surface defect detection method based on the dual-layer SAM model collaboration provided in the application constructs a dual-layer collaborative architecture in which an upstream SAM is used for semantic segmentation and a downstream SAM is used for abnormality refinement, and combines an adaptive feature converter and a core memory library optimization strategy to achieve high-precision defect detection in a small sample scenario.

[0053] Other technical effects brought by the additional features will be further described in the corresponding embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0054] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0055] Figure 1 A flowchart of the PCB surface defect detection method based on dual-layer SAM model collaboration according to an exemplary embodiment is shown.

[0056] Figure 2 A structural diagram of the PCB surface defect detection system based on dual-layer SAM model collaboration according to an exemplary embodiment is shown.

[0057] Figure 3 A schematic diagram of the dual-layer SAM collaboration mechanism according to an exemplary embodiment is shown.

[0058] Figure 4 An adaptive feature converter structure diagram shown according to an exemplary embodiment;

[0059] Figure 5 A core memory library construction flow chart shown according to an exemplary embodiment;

[0060] Figure 6 An abnormal score fusion strategy diagram shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0061] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. The parts not described in detail in the following embodiments can be implemented by using existing technologies.

[0062] The PCB defect detection technology still has obvious deficiencies in the core technical dimensions such as small sample learning adaptability, semantic information mining depth, and small defect segmentation precision. Based on the above problems, the embodiment of the present application provides a PCB surface defect detection method based on double-layer SAM model cooperation to solve the above problems.

[0063] Referring to Figure 1 and Figure 3 , in an embodiment of the present application, a PCB surface defect detection method based on double-layer SAM model cooperation includes:

[0064] S100, using a pre-trained ResNet18 model to extract natural image domain features of a to-be-tested PCB image, and converting the natural image domain features into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information.

[0065] S200, using an upstream SAM model to perform global semantic segmentation on the to-be-tested PCB image, extracting a plurality of semantic masks, and representing the plurality of semantic masks as weighted semantic mask features;

[0066] S300, based on the image block features and the weighted semantic mask features, calculating the Euclidean distance between the to-be-tested PCB image and a core memory library constructed based on normal PCB images to obtain multi-level abnormal scores;

[0067] S400, fusing the multi-level abnormal scores to obtain a rough abnormal score map;

[0068] S500, using a downstream SAM model to obtain an effective segmentation region mask by taking the rough abnormal score map as a prompt and combining the normal PCB structure learned by the upstream SAM model.

[0069] S600, performing pixel-level abnormal region refinement on the coarse abnormal score map based on the effective segmentation region mask to obtain a fine abnormal score map, for category judgment and positioning of defect detection.

[0070] In an embodiment of the present application, a double-layer collaborative architecture is constructed, in which an upstream SAM is used for semantic segmentation and a downstream SAM is used for abnormality refinement, combined with an adaptive feature converter and a core memory bank optimization strategy, to realize high-precision defect detection in a small sample scenario.

[0071] In some specific embodiments of the present application, S100, a pre-trained ResNet18 model is used to extract natural image domain features of the PCB image to be tested, and an adaptive feature converter is used to convert the features into industrial image domain features, and an image block feature containing neighborhood information is constructed, which can be represented as:

[0072] S101 uses a pre-trained ResNet18 to extract multi-level features of the PCB image, φi,j∈R (Cj×Hj×Wj) , where j∈[2,3]. J means level, and i represents a pixel.

[0073] Specifically, the middle layer (j=2) and high layer (j=3) features of ResNet18 are focused on, which not only retains the local detail information of the middle layer features (such as pad edge texture, trace width), but also fuses the semantic structure information of the high layer features (such as the connection relationship of "pad-trace"), realizing multi-level feature coverage from pixel-level details to structure-level semantics, and providing more comprehensive feature support for subsequent defect detection.

[0074] Specifically, based on the lightweight architecture of ResNet18 and the targeted feature conversion, the feature expression ability is guaranteed while the computational complexity is controlled, avoiding the efficiency loss caused by deep networks, and laying a foundation for efficient features for subsequent memory bank construction and real-time detection.

[0075] S102 converts the natural image domain features into industrial image domain features through the adaptive feature converter Fθ .

[0076] Specifically, the adaptive feature converter is used to convert the natural image features of the pre-trained model, effectively alleviating the "distribution offset problem" of the pre-trained model on industrial PCB images (such as the significant difference between the low texture and high contrast characteristics of PCB images and natural images), making the extracted features more consistent with the visual properties of industrial structures such as pads and traces, and improving the feature's ability to express the PCB scene.

[0077] S103 constructs an image block feature representation containing neighborhood information .

[0078] Specifically, by constructing image patch features that include neighborhood information, the features of a single image patch are associated with the features of the surrounding area, effectively capturing the spatial spread characteristics of defects (such as the continuous distribution of scratches and the regional correlation of short-circuit defects), avoiding "local misjudgment" caused by isolated analysis of single patch features, and improving the feature representation ability of structural defects (such as broken wires and short circuits).

[0079] The embodiments described above in this application, through S101 comprehensive feature extraction, S102 domain adaptive optimization, and S103 spatial correlation modeling, achieve high-precision, highly adaptable, and highly efficient PCB defect detection support capabilities. This not only improves the feature representation accuracy of various defects but also ensures the real-time performance and robustness required for industrial scenarios, providing a more reliable feature foundation for subsequent defect detection tasks (such as classification and localization).

[0080] In one specific embodiment, the transformation process of S101-S103 in the feature domain can be implemented using the following computer programming languages:

[0081] class AdaptiveFeatureTransformer(nn.Module):

[0082] def __init__(self, input_dim, hidden_dim=256):

[0083] super().__init__()

[0084] self.encoder = nn.Sequential(

[0085] nn.Conv2d(input_dim, hidden_dim, 3, padding= 1),

[0086] nn.ReLU(),

[0087] nn.Conv2d(hidden_dim, input_dim, 3, padding= 1) )

[0089] def forward(self, x):

[0090] return x + self.encoder(x) # Residual join

[0091] In some specific embodiments of this application, the aforementioned pre-trained ResNet18, such as... Figure 4 As shown, the following training method is used:

[0092] First, the normal PCB image features are injected into the industrial image domain features after Conv2D-ReLU, Conv2D residual connection Generate simulated abnormal features by injecting Gaussian noise: Where ε ~ N(0, σ2) represents Gaussian noise, φi- represents abnormal features, represents normal PCB images;

[0093] Specifically, traditional anomaly detection often relies on a large number of real abnormal samples, but abnormal samples (such as short circuit, open circuit, stain, etc.) in PCB production are often extremely small in number and varied in type, making it difficult to collect. This embodiment directly uses the normal sample features of normal PCB images Inject Gaussian noise (ε ~ N(0, σ²)) to generate simulated abnormal features (φi-), without real abnormal data to build a "normal-abnormal" contrast sample pair. This approach cleverly bypasses the pain point of the scarcity of abnormal samples, especially avoiding the cost of manually labeling abnormal samples.

[0094] Then, the discriminator Du is used to evaluate the difference between the normal sample features and the abnormal features φi-;

[0095] Finally, the converter parameters θ are optimized through adversarial training Specifically, through the adversarial game (minθmaxψ objective function) of the discriminator Du and the converter, a closed loop of generating simulated abnormalities, discriminators, and converter optimization feature expression is realized: the discriminator Du needs to distinguish the normal features and the simulated abnormal features as much as possible, and promote its sensitivity to the difference between normal / abnormal; the converter parameters θ then indirectly optimize the feature space expression by minimizing the discriminant ability of the discriminator (i.e. make it more difficult to identify simulated abnormalities) - make the normal features more pure, and the difference between the abnormal features (even simulated) and the normal features more significant.

[0096] In a specific embodiment, the training of ResNet18 described above can be implemented using the following computer program language:

[0097] # Use pre-trained ResNet18

[0098] backbone = ResNet18(pretrained=True)

[0099] features = backbone.extract_features(input_image, layers= [2, 3]).

[0100] ​It is worth noting that in the above training process, noise injection and adversarial training are directly performed in the feature space (rather than the original image), which is more consistent with the working logic of deep learning models: models ultimately rely on features rather than raw pixels for judgment, and strengthening the differences between normal and abnormal at the feature level can more efficiently improve the model's perception of abnormalities.

[0101] The above embodiments of the present application, under the constraint of the scarcity of abnormal samples, actively construct the distinction between normal and abnormal through adversarial generation at the feature level, not only solving the data problem, but also specifically strengthening the model's perception of abnormal features, which is particularly suitable for the high-precision detection needs of small and unknown abnormalities in PCB and other industrial inspection scenarios.

[0102] In order to make the model focus more accurately on key structures, while adapting to the actual needs of industrial detection. In some specific embodiments of the present application, S200, an upstream SAM model is used to perform global semantic segmentation on the PCB image to be tested, extract multiple semantic masks, and represent them as weighted semantic mask features, as shown in Figure 3 The following steps can be used:

[0103] S201, after the SAM model receives the PCB image to be tested, it does not need to rely on manually labeled structure labels, but only needs to automatically group the pixels in the image by analyzing their color, texture, and shape similarity to obtain semantic masks, including 9 types of PCB structures.

[0104] For example, the 9 types of PCB structures include solder joints, conductive circuits, chip pins, pads, vias, solder masks, silk screen layers, edge blank areas, and non-functional markers.

[0105] S202, according to the importance of different PCB structures, define corresponding weight values, which reflect the importance of different regions (structures).

[0106] Specifically, the weight values here can be set according to experience.

[0107] S203, weight the 9 semantic masks with the corresponding weight values to obtain weighted semantic mask features.

[0108] Specifically, the abnormalities (such as short circuit, virtual welding, fracture) of the key structures (such as welding points, conductive lines, chip pins, etc.) may directly lead to the failure of the PCB function, and are the core concern objects of detection. The abnormalities of the secondary structures (such as edge blank area, non-functional mark, slight stain area, etc.) have less influence on the performance of the PCB, and even may be irrelevant interference (such as dust, shooting reflection). If the semantic mask features are not weighted, the model may consider all structure features as equally important, which may lead to: the abnormal features of the key structures are "diluted" by the normal features of the secondary structures, reducing the detection sensitivity; the slight interference of the secondary structures is misjudged as abnormal, increasing the false detection rate. The essence of weighting is to strengthen the contribution of the key structure features and weaken the interference of the secondary structures by using artificial or automatic defined weights, so that the model focuses more on the "areas that really need to be concerned" in subsequent detection.

[0109] In the above embodiments of the present application, steps S201-S203 adopt the combination of "SAM automatic segmentation + weighting", which reduces the threshold of industrial landing. Specifically, this method does not require manual annotation of structure labels, can adapt to the scene of various PCB models, and thus improves the flexibility and efficiency of the detection process. At the same time, this combination also improves the accuracy and robustness of detection. Through the weighting operation, the abnormal signals of the key structures are strengthened, and the secondary interference is suppressed, so that the model can more accurately identify "the abnormalities that really affect the function of the PCB", reducing the occurrence of missed detection and false detection.

[0110] In a specific embodiment of the present application, the above process of upstream SAM semantic segmentation can be implemented using the following computer program language:

[0111] # Use SAM for global segmentation

[0112] sam_predictor = SAM(checkpoint_path="sam_vit_h.pth")

[0113] masks = sam_predictor.predict_masks(image, n_masks=9)

[0114] # Calculate mask weights

[0115] weights = compute_mask_weights(masks, features) weighted_features =features weights

[0116] In order to make the feature distribution in the core memory library representative and diverse, in some specific embodiments of the present application, the core memory library is constructed, such asFigure 5 The following steps can be taken:

[0117] First, build the hybrid memory library where Mpatch is the image block level feature, and Mmask is the mask level feature.

[0118] Specifically, Mpatch (image block level feature): the features extracted for each small block after the PCB image is divided into multiple local small blocks (such as texture, color, local structure features). Such features have fine granularity and can reflect local details of the image, such as minor deformation of the solder joint and local fracture of the circuit.

[0119] Mmask (mask level feature): features extracted based on the semantic mask mentioned above (such as the mask of 9 kinds of PCB structures), representing the overall features of the entire structure area, such as the overall contour of the chip area and the global distribution of the circuit network. Such features have coarse granularity and focus on structured information.

[0120] A single feature cannot balance "local details" and "global structure". For example, using only block features may ignore the relevance between structures, and using only mask features may lose the details of minor abnormalities. M integrates both types of features, capturing both local abnormal fine features and global structure information, providing a more comprehensive feature basis for subsequent abnormality judgment.

[0121] Next, use a greedy algorithm to optimize the target to achieve downsampling. Specifically, the formula represents: for each feature m in M, find the nearest feature n in the candidate subset Mc (distance measured by L2 norm ); calculate the maximum value of these "nearest distances" (i.e. ), which reflects the "maximum coverage error" of Mc to M - that is, the distance between the farthest feature in the original feature and the nearest element in Mc; find the subset Mc that minimizes this "maximum coverage error" (i.e. argmin), that is, let Mc "uniformly cover" the distribution of the original feature M as much as possible, and Mc is a subset of M.

[0122] Therefore, the subset obtained by the greedy algorithm finally makes each feature in the hybrid memory library find a most similar feature in the subset, and the Euclidean distance between each feature in the hybrid memory library and the most similar feature is minimized.

[0123] Of course, in order to make the core memory library obtained by downsampling further representative and diverse. In some specific embodiments, it is subjected to the following constraints:

[0124] The downsampling ratio is fixed at 1%, ensuring computational efficiency;

[0125] Ensure that the compressed features cover more than 95% of the original feature space;

[0126] A cluster center selection strategy is adopted to prioritize the retention of representative features.

[0127] The above embodiments of this application combine three constraints to significantly reduce the computational load of the core memory library by a 1% downsampling ratio, ensuring the real-time efficiency of industrial detection. Furthermore, by covering more than 95% of the original feature space and using a cluster center selection strategy to prioritize the retention of representative features, key feature loss is effectively avoided. Thus, while controlling computational costs, the feature distribution of the core memory library is both broadly representative and retains sufficient diversity, providing reliable support for the accuracy and robustness of subsequent defect detection.

[0128] In one specific implementation, the construction of the aforementioned core memory library can be achieved using the following computer programming languages:

[0129] # Building a hybrid memory library

[0130] patch_features=extract_patch_features(normal_images)

[0131] mask_features=extract_mask_features(normal_images,sam_masks) hybrid_memory=torch.cat([patch_features,mask_features],dim=0)

[0132] The greedy downsampling algorithm used can be implemented in the following computer programming languages:

[0133] def greedy_subsampling(memory_bank, subsample_ratio=0.01):

[0134] n_samples = int(len(memory_bank) subsample_ratio)

[0135] selected_indices = [0] # Initially select the first sample

[0136] for _ in range(n_samples - 1):

[0137] distances = [ ]

[0138] for i in range(len(memory_bank)):

[0139] if i is in selected_indices:

[0140] continue

[0141] # Calculate the minimum distance to the selected samples

[0142] min_dist = min([torch.norm(memory_bank[i] - memory_bank[j])

[0143] forj in selected_indices])

[0144] distances.append((min_dist, i))

[0145] # Select the sample with the largest distance

[0146] best_idx = max(distances)

[0147] selected_indices.append(best_idx)

[0148] return memory_bank[selected_indices]

[0149] Based on the core memory library established in the above embodiments, in some specific implementations of this application, in S300, based on image block features and weighted semantic mask features, the Euclidean distance between the PCB image to be tested and the core memory library constructed based on the normal PCB image is calculated to obtain a multi-level anomaly score, such as... Figure 6 As shown, the following steps can be taken:

[0150] S301, calculate the Euclidean distance between the image block features of the PCB image under test and all image feature blocks in the core memory library; take the minimum distance as the initial anomaly metric for that image block. An adaptive weighting mechanism is used to calculate the weights: , Let m be the weight, and m be the image patch features in the core memory library. test These are the image block features of the PCB image under test. It is the initial anomaly measurement. Indicates that in the kernel memory library and The neighboring feature set. The weight is fused with the initial abnormality metric to obtain an image patch level score sp.

[0151] S302, similarly, the same calculation steps as 301 are adopted to obtain the initial abnormality metric and the weight, and the mask level score is calculated .

[0152] Further, the two abnormality scores are fused. In some embodiments of the present application, S400, the multi-level abnormality scores are fused to obtain a coarse abnormality score map, which can adopt the following steps: fusing the image patch level score sp and the mask level score .

[0153] Specifically, the image patch level score sp: the abnormality degree score of the image patch. It focuses on local details and can capture small local abnormalities, such as a single solder joint virtual welding or a slight fracture of a certain line, but it may be interfered by local noise, such as isolated pixel abnormalities. The mask level score : the abnormality degree score based on the semantic mask (such as the overall structure of the "chip area" and "line network"). It reflects the global structure abnormality (such as the overall shape abnormality of a certain structure or the absence of a key structure), but it has low sensitivity to small local abnormalities.

[0154] The fusion method of "taking the maximum value" is essentially "not missing any possible abnormal signal": when the image patch level score sp is high (there is obvious abnormality in the local), even if the mask level score is low (the global structure looks normal as a whole), the final score will also take sp, ensuring that small local abnormalities are not ignored; when the mask level score is high (there is significant abnormality in the global structure), even if the local patch level score sp is low (no obvious abnormality is found in the local details), the final score will take , avoiding missing the global structural abnormality (such as the overall shift of the chip).

[0155] PCB abnormalities can be both small local defects (such as pinholes and small area corrosion) and global structural problems (such as overall misplacement of lines and missing of components). The max function in the above embodiments of the present application ensures that abnormalities of both scales can be captured, avoiding the one-sidedness of a single scale score.

[0156] For the case of "local abnormality but global structure looks normal" (such as a small fracture of a certain line that does not affect the overall structure form), the high value of sp will dominate the final score to prevent missed detection; for the case of "global abnormality but local details do not trigger a high score" , the high value of will dominate the score, also avoiding missed detection.

[0157] The operation of taking the maximum is simple and efficient, without complex weight calculation, and can quickly output the fusion result while ensuring coverage of various abnormalities, meeting the needs of "high sensitivity + high efficiency" in industrial detection.

[0158] Early fusion score Although the information of local blocks and global masks is taken into account, there are two potential limitations: boundary ambiguity problem: the image block-level score sp may give an ambiguous score (neither high nor low) at the edge of the abnormal area, and the mask-level score Focus on the overall structure, it is difficult to distinguish the boundary details; semantic misjudgment risk: the features of some local areas may be similar to the abnormality (such as local brightness abnormality caused by shooting reflection), but from the global semantic point of view, it is not a real abnormality (such as reflection in a non-key area), and the early fusion score may misjudge as an abnormality. Therefore, in some specific embodiments of the present application, S500, a downstream SAM model is used to obtain an effective segmentation area mask by using the rough abnormal score map as a prompt and combining the normal PCB structure learned by the upstream SAM model. The following steps can be used:

[0159] S501, select the pixel point with the highest score in the rough abnormal score map as a positive prompt point.

[0160] Specifically, the model attention is directly guided to the core area that is "most likely to be abnormal", avoiding the downstream SAM searching in the whole image without difference, reducing invalid calculation, and improving the segmentation efficiency.

[0161] S502, input the rough abnormal score map as a mask prompt into the downstream SAM model.

[0162] Specifically, it is equivalent to giving the downstream SAM a "preliminary range guide", so that it performs more detailed segmentation within the range, rather than starting from scratch, greatly improving the relevance and accuracy of segmentation.

[0163] S503, the downstream SAM model generates multiple initial segmentation areas based on the positive prompt point and the mask prompt, and combines the normal PCB structure semantic information learned by the upstream SAM model.

[0164] S504, filter the initial segmentation areas, and remove the segmentation results whose coverage area ratio exceeds the threshold β = 0.2.

[0165] Specifically, by threshold constraint, large-area regions that do not meet the "local abnormality" feature are filtered out, reducing false positive labeling (such as misjudging the slight brightness difference of the whole image as an abnormality), and ensuring that the remaining segmentation results are "small range, high risk" true abnormalities.

[0166] S505, the segmentation results reserved after the above filtering jointly constitute an effective segmentation region mask.

[0167] It is worth noting that the upstream SAM model has learned the structural semantics of normal PCBs through the previous steps, such as the normal direction of the lines, the typical shape of the solder joints, and the relative position relationship of each structure. Therefore, the upstream SAM model provides a normal reference here: when segmenting, the downstream SAM compares the "current region features" with the "normal structure features learned by the upstream" to determine whether the region deviates from the normal form, avoiding misjudgment of "normal but special form structures as abnormal. Of course, the upstream SAM model can constrain the rationality of segmentation here: PCB is a highly structured product, and the form, size, and position of each component have fixed logic (such as uniform line width, and close connection between solder joints and lines). The downstream SAM uses these normal semantics to correct the "false abnormal signals" in the rough abnormal score map, such as local deformation caused by the shooting angle, which actually belongs to the normal structure, so that the segmentation result is more consistent with the physical structure logic of the PCB.

[0168] In order to more accurately support the category judgment and positioning of PCB defects, some specific embodiments of the present application perform pixel-level abnormal region refinement on the rough abnormal score map based on the effective segmentation region mask to obtain a fine abnormal score map for category judgment and positioning of defect detection, which can use the following steps:

[0169] The rough abnormal score map and the enhancement item are superimposed to obtain a fine abnormal score map; wherein the enhancement item is the product of the enhancement coefficient, the effective segmentation region mask, and the rough abnormal score Figure Three . Wherein a = 0.3 is the enhancement coefficient.

[0170] The above embodiments of the present application can accurately amplify the effective signal and accurately suppress the invalid interference. By incorporating the prior information of the effective segmentation region into the score calculation, the finally generated fine abnormal score map can not only retain the positioning accuracy at the pixel level, but also strengthen the distinction between the defect region and the normal region. This provides a high-quality feature basis for the accurate positioning and accurate classification of printed circuit board (PCB) defects, which perfectly meets the core requirements of "high precision and low misjudgment" in industrial quality inspection.

[0171] In a specific embodiment, the process of detailed implementation in the test phase can be implemented using the following computer program languages:

[0172] Multi-level abnormal score calculation:

[0173] Image block-level score:

[0174] python

[0175] def compute_patch_scores(test_features, core_memory):

[0176] patch_scores = [ ]

[0177] for patch_feat in test_features:

[0178] distances = torch.norm(patch_feat - core_memory, dim= 1) min_distance= torch.min(distances)

[0179] patch_scores.append(min_distance) return torch.stack(patch_scores)

[0180] Mask-level scores:

[0181] def compute_mask_scores(test_image, sam_masks, core_memory): mask_scores = [ ]

[0182] for mask in sam_masks:

[0183] mask_feat = extract_mask_feature(test_image, mask)

[0184] distances = torch.norm(mask_feat - core_memory, dim= 1) min_distance= torch.min(distances)

[0185] mask_scores.append(min_distance) return torch.stack(mask_scores)

[0186] Score fusion:

[0187] def fuse_scores(patch_scores, mask_scores):

[0188] # Up-sample to the same size

[0189] patch_map = F.interpolate(patch_scores, size=(H, W))

[0190] mask_map = F.interpolate(mask_scores, size=(H, W))

[0191] # Take max fusion

[0192] fused_map = torch.max(patch_map, mask_scores)

[0193] return fused_map

[0194] Downstream SAM refinement

[0195] def downstream_sam_refinement(test_image, coarse_map, alpha=0.3):

[0196] # Select high score points as prompts

[0197] high_score_points = select_high_score_points(coarse_map, top_k=5)

[0198] # SAM refinement

[0199] refined_mask = sam_predictor.predict( image=test_image,

[0200] point_prompts=high_score_points,

[0201] mask_prompt =coarse_map > threshold )

[0202] # Result fusion

[0203] final_map = coarse_map + alpha refined_mask coarse_map returnfinal_map

[0204] Based on the same technical concept, in other embodiments of the present application, a PCB surface defect detection system 100 based on a double-layer SAM model collaboration, like Figure 2As shown, comprising:

[0205] The feature extraction module 110 extracts the natural image domain features of the to-be-tested PCB image using a pre-trained ResNet18 model, and converts them into industrial image domain features through an adaptive feature converter, to construct image block features containing neighborhood information.

[0206] The upstream module 120 adopts an upstream SAM model to perform global semantic segmentation on the to-be-tested PCB image, extracts a plurality of semantic masks, and represents them as weighted semantic mask features.

[0207] The scoring module 130 calculates the Euclidean distance between the to-be-tested PCB image and the core memory library constructed based on normal PCB images based on the image block features and the weighted semantic mask features, to obtain multi-level abnormality scores.

[0208] The fusion module 140 fuses the multi-level abnormality scores to obtain a coarse abnormality score map.

[0209] The downstream module 150 adopts a downstream SAM model to obtain an effective segmentation region mask by taking the coarse abnormality score map as a prompt and combining the normal PCB structure learned by the upstream SAM model.

[0210] The refinement module 160 performs pixel-level abnormal region refinement on the coarse abnormality score map based on the effective segmentation region mask to obtain a fine abnormality score map, which is used for category judgment and positioning of defect detection.

[0211] The modules / units in the above examples of the present application can specifically refer to the implementation techniques of the corresponding steps of the PCB surface defect detection method based on the double-layer SAM model cooperation in the above embodiments, which will not be repeated here.

[0212] The above technical solutions of the present application will be further described in conjunction with specific application examples / contrastive examples, so that the above technical solutions of the present application can be better understood. It should be understood that the following are only some examples and do not limit the present application.

[0213] The key parameter configuration and optimization of the PCB surface defect detection method based on the double-layer SAM model cooperation in the above embodiments are shown in Table 1.

[0214]

[0215] Application scenario 1 is PCB trace defect detection. For trace defect optimization parameters, increase the number of masks to 12 to capture slender traces, use feature layers 1, 2, and 3; and enhance the reinforcement coefficient to 0.5.

[0216] The application scenario 2 is element missing detection. For element missing optimization parameters, the number of masks is increased to 12 to capture element regions, the larger patch size is 3.2, and the slightly larger memory library'memory_ratio' is 0.02.

[0217] The above application verifies the feasibility of the method of the application.

[0218] In order to verify the technical effect of the method of the embodiment of the application, comparative experiments are carried out. The experiments are based on the MVTecAD standard dataset, the self-built 10-class defect PCB dataset and the small sample test set (1-shot, 5-shot), and the classification AUROC, segmentation AUROC and other indicators are verified, and the comparison results are shown in Table 2:

[0219]

[0220] As shown in Table 2, the classification and segmentation AUROC of the method of the application are significantly ahead, the generalization of the small sample scene (1-shot reaches 88.3%, 5-shot reaches 92.1%) and the detection speed (22.7FPS) are obvious.

[0221] In order to verify the gain of each module component on the precision, the ablation experiment is carried out, and the results are shown in Table 3.

[0222]

[0223] As can be seen, the ablation experiment verifies that each component is useful and can produce gain.

[0224] In the description of the embodiments of the application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the devices or elements indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application.

[0225] In addition, the terms "first", "second" are only for description purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features.

[0226] In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified and limited. In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood in a broad sense, for example, can be fixedly connected, can also be detachably connected, or integrally connected; can be mechanically connected, can also be electrically connected; can be directly connected, can also be indirectly connected through an intermediate medium, can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0227] In the embodiments of the present application, the terms "including" and "having" and any variants thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally further comprises steps or units not listed, or optionally further comprises other steps or units inherent to the process, method, product or device.

[0228] The above describes some specific embodiments of the present application. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or modifications within the scope of the claims, which does not affect the essential content of the present application. The above preferred features can be used in combination in the case of not conflicting with each other.

Claims

1. A PCB surface defect detection method based on a double-layer SAM model cooperation, characterized in that, The method comprises the following steps: extracting the natural image domain features of the to-be-tested PCB image using a pre-trained ResNet18 model, and converting them into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information; using an upstream SAM model to perform global semantic segmentation on the to-be-tested PCB image, extract multiple semantic masks, and represent them as weighted semantic mask features; based on the image block features and the weighted semantic mask features, calculating the Euclidean distance between the to-be-tested PCB image and the core memory library constructed based on normal PCB images to obtain multi-level anomaly scores; fusing the multi-level anomaly scores to obtain a coarse anomaly score map; using a downstream SAM model to obtain an effective segmentation region mask based on the coarse anomaly score map and the normal PCB structure learned by the upstream SAM model; based on the effective segmentation region mask, performing pixel-level anomaly region refinement on the coarse anomaly score map to obtain a fine anomaly score map for class judgment and positioning of defect detection.

2. The PCB surface defect detection method based on the double-layer SAM model coordination according to claim 1, characterized in that, The training process of the adaptive feature converter comprises: injecting Gaussian noise into the industrial image domain features of normal PCB images to generate simulated abnormal features; using a discriminator to evaluate the difference between normal industrial image domain features and the abnormal features; optimizing the parameters of the adaptive feature converter through adversarial training.

3. The PCB surface defect detection method based on the double-layer SAM model coordination according to claim 1, characterized in that, The method of using an upstream SAM model to perform global semantic segmentation on the to-be-tested PCB image, extract multiple semantic masks, and represent them as weighted semantic mask features comprises: After the SAM model receives the to-be-tested PCB image, it does not rely on manually labeled structure labels, but automatically groups the pixels in the image based on color, texture, and shape similarity to obtain 9 semantic masks containing 9 types of PCB structures; define corresponding weight values according to the importance of different PCB structures; multiply the 9 semantic masks by the corresponding weight values to obtain weighted semantic mask features.

4. The PCB surface defect detection method based on the double-layer SAM model coordination according to claim 1, characterized in that, The construction process of the core memory library constructed based on normal PCB images comprises: using the image block features and the weighted semantic mask features of normal PCB images to form a hybrid memory library; using a greedy algorithm to find a subset such that each feature in the hybrid memory library can find a most similar feature in the subset, and the Euclidean distance between each feature in the hybrid memory library and the most similar feature is minimized; the subset is used as the core memory library.

5. The PCB surface defect detection method based on the double-layer SAM model coordination according to claim 4, characterized in that, The core memory library satisfies one or more of the following constraints: the proportion of the number of features of the subset after downsampling to the total number of features of the hybrid memory library is fixed at 1%; the features in the subset cover more than 95% of the original feature space of the hybrid memory library; the features in the subset are selected using a cluster center selection strategy.

6. The PCB surface defect detection method based on the double-layer SAM model coordination according to claim 1, characterized in that, The method of calculating the Euclidean distance between the to-be-tested PCB image and the core memory library constructed based on normal PCB images based on the image block features and the weighted semantic mask features to obtain multi-level anomaly scores comprises: For the image block feature, the Euclidean distance with all image block features of the core memory library is calculated, and the minimum distance is taken as the initial anomaly measure; the image block feature weight is calculated; the image block level score is obtained based on the initial anomaly measure and the image block feature weight; For the weighted semantic mask feature, the Euclidean distance with all weighted semantic mask features of the core memory library is calculated, and the minimum distance is taken as the initial anomaly measure; the weighted semantic mask feature weight is calculated; the mask level score is obtained based on the initial anomaly measure and the weighted semantic mask feature.

7. The PCB surface defect detection method based on the double-layer SAM model coordination according to claim 1, characterized in that, The multi-level anomaly scores are fused to obtain a rough anomaly score map, including: The image block level score and the mask level score are compared, and the maximum value of the two is taken as the final fusion score.

8. The PCB surface defect detection method based on the double-layer SAM model coordination according to claim 1, characterized in that, The downstream SAM model is used, and the rough anomaly score map is used as a prompt, combined with the normal PCB structure learned by the upstream SAM model, to obtain an effective segmentation region mask, including: Select the pixel point with the highest score in the rough anomaly score map as the positive prompt point; The rough anomaly score map is input into the downstream SAM model as a mask prompt; The downstream SAM model generates multiple initial segmentation regions based on the positive prompt point and the mask prompt, combined with the normal PCB structure semantic information learned by the upstream SAM model; The initial segmentation regions are filtered, and the segmentation results with an area coverage ratio exceeding a threshold are removed; The segmentation results remaining after the above filtering jointly constitute an effective segmentation region mask.

9. The PCB surface defect detection method based on the double-layer SAM model coordination according to claim 1, characterized in that, Based on the effective segmentation region mask, pixel-level anomaly region refinement is performed on the rough anomaly score map to obtain a fine anomaly score map, which is used for category judgment and positioning of defect detection, including: The rough anomaly score map is superimposed with an enhancement item to obtain a fine anomaly score map; wherein the enhancement item is the product of an enhancement coefficient, the effective segmentation region mask and the rough anomaly score map.

10. A PCB surface defect detection system based on the synergistic effect of the double-layer SAM model, characterized by, Including: The feature extraction module: using a pre-trained ResNet18 model to extract the natural image domain features of the to-be-tested PCB image, and converting them into industrial image domain features through an adaptive feature converter to construct image block features containing neighborhood information; The upstream module: using an upstream SAM model to perform global semantic segmentation on the to-be-tested PCB image, extract multiple semantic masks, and represent them as weighted semantic mask features; The scoring module: based on the image block features and the weighted semantic mask features, the Euclidean distance between the to-be-tested PCB image and the core memory library based on normal PCB images is calculated to obtain multi-level anomaly scores; The fusion module: fusing the multi-level anomaly scores to obtain a rough anomaly score map; The downstream module: using a downstream SAM model, taking the rough anomaly score map as a prompt, and combining the normal PCB structure learned by the upstream SAM model to obtain an effective segmentation region mask; The refinement module: based on the effective segmentation region mask, performing pixel-level anomaly region refinement on the rough anomaly score map to obtain a fine anomaly score map, which is used for category judgment and positioning of defect detection.

Images