A small sample PCBA component target detection method and system based on a visual base model

Abstract

This invention belongs to the field of PCBA quality inspection and relates to a method and system for small-sample PCBA component target detection based on a visual fundamental model. The method includes: obtaining pre-positioning prompt boxes from the PCBA image to be detected using a localization fundamental model; obtaining single-scale frozen features of the image using a classification fundamental model; expanding the single-scale features into multi-scale features for pre-positioning and classification tasks; filtering and selecting the pre-positioning prompts; and obtaining query point features using a feature-aligned query sampling method; predicting classification and regression results for each query point feature based on the multi-scale localization features, classification features, query point features, and prompt boxes. This invention enables rapid fine-tuning of a high-precision model even with scarce training data, reducing training costs and effectively addressing the need for rapid production line changeover in small-batch PCBA production.

Description

Technical Field

[0001] This invention belongs to the field of PCBA quality inspection technology, and more specifically, relates to a small sample PCBA component target detection method and system based on a visual basic model, for PCBA components. Background Technology

[0002] Automatic Optical Inspection (AOI) technology is widely used in industrial product quality inspection due to its advantages such as non-contact operation, high speed, and high precision. The theoretical basis of AOI technology comes from machine vision, and the inspection process mainly includes four steps: visual imaging, region of interest (ROI) localization, algorithm processing, and post-processing. The most critical step is the algorithm processing stage, which involves three main modules: the computing platform, the detection algorithm, and the control interface. Target detection algorithms are typically used for pre-locating key areas, such as the localization and identification of components on PCBA products and the localization of chip unit markers.

[0003] Defect detection of PCBA components is a crucial step in PCBA quality inspection. The common method involves using target detection to pinpoint the specific location of components and classify them, obtaining the RoI (Region of Interest) for each component and initially identifying defects such as missing or incorrect components. Subsequently, classification or anomaly detection methods are used to individually assess whether each component contains defects. Good component location and identification performance is of great significance for subsequent component analysis, helping to optimize PCBA production processes and material selection, improve production line stability and product quality, and reduce production costs.

[0004] Although object detection is increasingly being used in the inspection of PCBA components, it usually requires long-term pre-training using large-scale, finely labeled datasets. Furthermore, the trained models can typically only handle the scenarios corresponding to the current training dataset. When the scenario changes (e.g., the detection object changes, or the imaging environment changes), the model will not be able to achieve the original performance and must be re-collected and retrained. This undoubtedly increases the difficulty of applying the algorithm in small-batch production scenarios.

[0005] In small-batch PCBA production, data acquisition and annotation time is drastically compressed to enable rapid deployment of detection algorithms in the early stages of production. Workers can only collect a very small amount of finely annotated training data. Supervised object detection algorithms suffer from severely reduced accuracy due to long fine-tuning times, poor generalization ability, and limited training samples, making them unsuitable for rapid deployment. Therefore, many existing methods investigate how to improve model detection accuracy in small-sample scenarios. Typical small-sample object detection algorithms include meta-learning-based and transfer learning-based algorithms.

[0006] Meta-learning-based few-shot object detection algorithms train the model in a class-independent manner. By grouping the training data and frequently switching the classifier's corresponding category, the model's sensitivity to specific categories is reduced, making the classification process rely only on the similarity between the features to be classified and the features of the reference image. This method is typically complex to train, slow to train, and heavily reliant on high-quality reference images, making it difficult to apply in the dynamic industrial scenarios.

[0007] Few-shot object detection algorithms based on transfer learning divide the model training process into two stages: pre-training and fine-tuning. In the pre-training stage, large-scale open-source natural image datasets (such as COCO and VOC) are used to obtain good feature representations. Then, the backbone network is frozen or its learning rate is reduced, and the detection head is trained normally. Fine-tuning is then performed using few-shot data. This method can reduce the damage to the high-quality features from large-scale pre-training by few-shot data during fine-tuning, thus ensuring the generalization ability of the trained model and improving the accuracy of few-shot learning. However, because industrial data and natural data domains typically have significant differences, models trained on natural data have poor generalization ability to PCBA images, affecting the model's fine-tuning accuracy. This leads to problems such as low component classification scores and inability to identify components, making them unsuitable for direct application in industrial scenarios.

[0008] To address the aforementioned issues, a small-sample target detection method that performs well in cross-domain scenarios needs to be proposed. This method can be rapidly fine-tuned using a very small amount of training data in small-batch PCBA production scenarios, adapting to rapid production line switching, and possessing high accuracy and high generalization. Summary of the Invention

[0009] To address the aforementioned deficiencies or improvement requirements of existing technologies, this invention provides a method and system for detecting small-sample PCBA components based on a visual fundamental model, thereby solving the problem of difficulty in generalizing features and accurately identifying components during fine-tuning in PCBA production scenarios.

[0010] To achieve the above objectives, according to one aspect of the present invention, a method for detecting small-sample PCBA components based on a visual fundamental model is provided, comprising the following steps: Offline pre-training: The PCBA image dataset with object detection bounding boxes is divided into multiple batches. The DINOv2 base model is used to extract single-scale features for each image in each batch. Then, the single-scale features are expanded into multi-scale localization features and classification features using a scale-expanding encoder and bilinear interpolation, respectively. The FastSAM base model is used to obtain the pre-localization bounding boxes for each image in each batch. Query point features and prompt boxes are obtained from the multi-scale localization features and classification features, respectively. The multi-scale localization features, classification features, query point features, and prompt boxes are simultaneously input into the task decoupled query decoder to predict the classification result and regression result for each query point feature. Calculate the object detection loss between the classification and regression results and the ground truth box, set the total number of iterations and the learning rate during the training process, freeze the base models DINOv2 and FastSAM, adjust the parameters in the scale expansion encoder and task decoupling query decoder to reduce the class-independent localization loss and object detection loss function, and the task decoupling query decoder corresponding to the preset total number of training iterations is the optimal task decoupling query decoder. Online fine-tuning: The PCBA image dataset used for fine-tuning is divided into multiple batches; the class-independent localization loss and object detection loss of the model training are obtained using the same steps as offline pre-training; FastSAM is fine-tuned using the convolutional low-rank fine-tuning module, and the class-independent localization loss between the pre-location box and the ground truth box is calculated; the total number of iterations and the learning rate are set during the training process, and the parameters in the convolutional low-rank fine-tuning module, the scale expansion encoder, the class-independent pre-location head, and the task decoupling query decoder are adjusted to reduce the class-independent localization loss and the object detection loss. When the preset total number of training iterations is reached, the corresponding task decoupling query decoder is the optimal task decoupling query decoder. Online reasoning: The PCBA images to be processed are input into DINOv2 to extract single-scale features for each image in each batch. Then, the single-scale features are expanded into multi-scale localization features and classification features using a scale-expanding encoder and bilinear interpolation, respectively. FastSAM is used to obtain the pre-localization bounding boxes for each image in each batch, and query point features and cue boxes are obtained from the multi-scale localization features and classification features, respectively. The multi-scale localization features, classification features, query point features, and cue boxes are simultaneously input into the task decoupled query decoder to predict classification and regression results for each query point feature. The high-scoring non-overlapping prediction boxes are selected as the target detection results.

[0011] Furthermore, the single-scale features are expanded into multi-scale localization features and classification features using a scale-extended encoder and bilinear interpolation. The scale-extended encoder aggregates the top-level features using self-attention, and then a multi-receptive field fusion module forms a dual-path fusion from top to bottom and bottom to top. The multi-receptive field fusion module uses a 3-layer multi-receptive field parallel structure to extract features and adds a residual structure to prevent gradient vanishing. The 3-layer multi-receptive field parallel structure includes a 1×1 convolution, a 3×3 convolution, and a cross-layer connection to process and add the features together.

[0012] Furthermore, bilinear interpolation is used to align the single-layer features extracted by DINOv2 with the original image in terms of size.

[0013] Furthermore, the FastSAM base model is fine-tuned using a low-rank convolution module; the low-rank convolution calculation process is represented as follows:

[0014] Where x represents the input feature, and This represents a low-rank convolutional layer, where W1 represents the number of channels that convert the features from the original number of channels. Reduce to 1×1 convolution, To restore the feature channels as A convolution with the same kernel size as the original convolution. The original convolution kernel, For bias, This represents the convolution operation. During the inference phase, the associative law of convolution is used to merge the low-rank convolution kernel with the original convolution kernel, ensuring that the model structure remains unchanged before and after fine-tuning. This process is represented as:

[0015]

[0016] in, The merged convolution kernel is compared with the original convolution kernel. They are the same size.

[0017] Furthermore, the parameters of the FastSAM base model are optimized through a self-supervised approach. This process first uses the untuned FastSAM model to predict targets on the image as pseudo-labels, and then merges the ground truth values ​​with the pseudo-labels as the actual labels used for fine-tuning training to maintain the performance of the original base model. The specific process is as follows:

[0018]

[0019] in, For the input image, This is the untuned FastSAM model. Indicates the post-processing procedure, including... Non-maximum suppression with threshold and Filter by score based on threshold. The pseudo-labels consist of high-confidence, non-repeating prediction boxes. This represents the training annotation set that combines ground truth and pseudo-labels.

[0020] Furthermore, a feature-aligned query sampling method is used to obtain reference location indices, cue reference boxes, and query features from multi-scale features of classification and localization respectively; the location indices, cue reference boxes, and query features of classification and localization tasks have a one-to-one correspondence. A multi-layered shared classification and localization detection head is used to predict bounding boxes, and the multi-layered results are jointly post-processed.

[0021] Furthermore, a reference location index and a suggested reference box are obtained through a query filtering mechanism, and the query features at the reference location index are obtained by aligning multi-scale feature maps; the specific process is as follows:

[0022]

[0023] in, For the input image, This indicates the multi-scale characteristics of flattening. This indicates the index of the query location on the feature map. The set, This represents the set of tooltip reference boxes corresponding to each location index; This indicates the corresponding feature found from multi-scale features based on the location index.

[0024] Furthermore, a task-decoupled query decoder is used to compute the classification and regression processes in parallel. The decoder input includes the query feature set, reference box set, and multi-scale features for each localization and classification task, and outputs the classification and bounding box regression results corresponding to each query index.

[0025] Furthermore, a deformable attention mechanism with shared reference points is employed to preserve some interactions between tasks; the specific process is as follows:

[0026]

[0027]

[0028] in, Indicates each query point The corresponding prompt box, and This represents the linear layer weights and biases used to offset the reference box coordinates. and These represent the location indices corresponding to the localization and classification tasks, respectively. Query characteristics, and This represents the Sigmoid activation function and its inverse function. This indicates the prompt point after the offset. Represents the features of the input image. This represents the features obtained through resampling. Indicates grid sampling operation, Indicates the number of reference points. MHA stands for Multi-Head Attention Operation, output This represents the features after deformable attention shared by the reference point. Indicates a category or location branch, Indicates a category branch, This indicates a branch to be located.

[0029] According to another aspect of the present invention, a small sample PCBA component target detection system based on a visual fundamental model is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement a small sample PCBA component target detection method based on a visual fundamental model as described in any of the preceding claims.

[0030] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art: 1. This invention combines a visual base model for localization and classification, and utilizes its highly generalized feature information to improve the fine-tuning quality of the algorithm in the case of small samples, improve the model's responsiveness to unknown data, accurately locate and identify PCBA components, and has a fast fine-tuning speed. It can accurately identify a specified type of component in a complex PCBA with only a very small number of training samples.

[0031] 2. This invention designs a low-rank convolutional module and a self-supervised fine-tuning method to fine-tune the localization model FastSAM with small samples, thereby increasing its ability to perceive new types of samples while maintaining its original zero-sample localization performance. This results in more accurate pre-positioning results for components on PCBAs and improves the recall rate of effective targets.

[0032] 3. This invention proposes a novel feature scale expansion encoding method, which encodes the single-layer features output by the base model DINOv2 into multi-layer features of different scales in a pyramid structure, and uses them to locate branches, introducing richer fine-grained information into the frozen global features and improving the localization quality of the network.

[0033] 4. This invention proposes a novel task-decoupled decoder that separates localization and classification tasks to prevent optimization conflicts arising between the two tasks with different feature granularity requirements during small-sample training, thereby improving training efficiency and localization and classification accuracy. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall network model constructed according to a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of a low-rank convolutional module constructed according to a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of a scale-extended encoder constructed according to a preferred embodiment of the present invention; Figure 4 This is a schematic diagram of a task-decoupled query decoder constructed according to a preferred embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the target detection effect of a small sample of a natural scene dataset constructed according to a preferred embodiment of the present invention; Figure 6 This is a schematic diagram of the testing effect of a small sample PCBA component constructed according to a preferred embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0036] like Figure 1 and Figure 2 As shown, this invention provides a small-sample PCBA component target detection method based on a visual fundamental model, which includes the following steps: Offline pre-training: Obtain any large-scale image dataset with labeled object detection boxes and divide it into multiple batches. Use the base model DINOv2 to extract single-scale features for each image in each batch. Expand these features into multi-scale localization and classification features using a scale-extended encoder and bilinear interpolation, respectively. Use the base model FastSAM to obtain pre-defined bounding boxes for each image in each batch. Utilize a feature-aligned query sampling method to extract query point features and cue boxes from the multi-scale localization and classification features, respectively. Simultaneously input the multi-scale localization features, classification features, query point features, and cue boxes into a task-decoupled query decoder to predict classification and regression results for each query point feature.

[0037] Calculate the object detection loss between the classification and regression results and the ground truth box, set the total number of iterations and the learning rate during the training process, freeze the base models DINOv2 and FastSAM, adjust the parameters in the scale expansion encoder and task decoupling query decoder to reduce the class-independent localization loss and object detection loss function, and the task decoupling query decoder corresponding to the preset total number of training iterations is the optimal task decoupling query decoder.

[0038] Online fine-tuning: A small-scale image dataset of PCBAs for fine-tuning is obtained and divided into multiple batches. The class-independent localization loss and object detection loss for model training are obtained using the same steps as for pre-training. FastSAM is fine-tuned using a convolutional low-rank fine-tuning module, and the class-independent localization loss between pre-localized boxes and ground truth boxes is calculated. The total number of iterations and the learning rate are set during training. Parameters in the convolutional low-rank fine-tuning module, scale-expanding encoder, class-independent pre-localization header, and task decoupling query decoder are adjusted to reduce the class-independent localization loss and object detection loss. The task decoupling query decoder corresponding to the preset total number of training iterations is the optimal task decoupling query decoder.

[0039] Online reasoning: The PCBA images to be processed are input into DINOv2 to extract single-scale features for each image in each batch. These features are then expanded into multi-scale localization and classification features using a scale-expanding encoder and bilinear interpolation, respectively. FastSAM is used to obtain pre-defined bounding boxes for each image in each batch. A feature-aligned query sampling method is used to extract query point features and cue boxes from the multi-scale localization and classification features, respectively. The multi-scale localization features, classification features, query point features, and cue boxes are simultaneously input into a task-decoupled query decoder to predict classification and regression results for each query point feature. High-scoring non-overlapping predicted bounding boxes are selected using object detection post-processing techniques.

[0040] More preferably, the scale-expanding encoder aggregates top-level features using self-attention, and then uses a multi-receptive-field fusion module to form a top-down and bottom-up dual-path fusion. The multi-receptive-field fusion module is a modified residual module that receives a cascade of low-level and high-level features, extracts features using a 3-layer multi-receptive-field parallel structure (including a 1×1 convolution, a 3×3 convolution, and a cross-layer connection to process and add features), and adds a residual structure to prevent gradient vanishing.

[0041] More preferably, bilinear interpolation is used to align the size of the single-layer features extracted by DINOv2 to 16 times the downsampled size of the original image.

[0042] More preferably, the method involves a low-rank convolutional module for fine-tuning the FastSAM base model. Its computation process is as follows:

[0043] Where x represents the input feature, W1 and W2 represent low-rank convolutional layers, where W1 is a 1×1 convolution that reduces the number of channels d to r, and W2 is a convolution with the same kernel size as the original convolution that restores the number of feature channels to d. W0 is the original convolution kernel, and b0 is the bias. This represents the convolution operation. During the inference phase, the associative law of convolution is used to merge the low-rank convolution kernel with the original convolution kernel, ensuring that the model structure remains unchanged before and after fine-tuning. This process can be represented as:

[0044]

[0045] Where W is the merged convolution kernel, which is the same size as the original convolution kernel W0.

[0046] More preferably, the fine-tuning of the FastSAM base model is performed through a self-supervised training method. This process first uses the untuned FastSAM model to predict targets on the image as pseudo-labels, then merges the ground truth values ​​with the pseudo-labels as the actual labels used for fine-tuning training, thus maintaining the performance of the original base model. Specifically, the process can be represented as follows:

[0047]

[0048] in For the input image, This is the untuned FastSAM model. Indicates the post-processing procedure, including... Non-maximum suppression with threshold and Filter by score based on threshold. The pseudo-labels consist of high-confidence, non-repeating prediction boxes. This represents the training annotation set that combines ground truth and pseudo-labels, used for actual fine-tuning training.

[0049] More preferably, the feature-aligned query sampling method obtains the reference location index, the cue reference box, and the query feature from the classification and localization multi-scale features, respectively. The location index, cue reference box, and query feature for the classification and localization task have a one-to-one correspondence.

[0050] More preferably, the feature-aligned query sampling method obtains the reference position index and the prompt reference box through a query filtering mechanism, and obtains the query features at the reference position index by aligning multi-scale feature maps. The specific process can be represented as follows:

[0051]

[0052] Where X is the input image, x represents the flattened multi-scale features, Q represents the set of query location indices q on the feature map, and R represents the set of cue reference boxes corresponding to each location index. This indicates the corresponding feature found from multi-scale features based on the location index.

[0053] More preferably, the task-decoupled query decoder performs parallel computation of the classification and regression processes. The decoder input includes the query feature sets, reference box sets, and multi-scale features for each of the localization and classification tasks, and outputs the classification and bounding box regression results corresponding to each query index.

[0054] More preferably, the task decoupling query decoder uses deformable attention with shared reference points to preserve some interactions between tasks. Specifically, the process is represented as follows:

[0055]

[0056]

[0057] in, Indicates each query point The corresponding prompt box, and This represents the linear layer weights and biases used to offset the reference box coordinates. and These represent the location indices corresponding to the localization and classification tasks, respectively. Query characteristics, and This represents the Sigmoid activation function and its inverse function. This indicates the prompt point after the offset. Represents the features of the input image. This represents the features obtained through resampling. Indicates grid sampling operation, Indicates the number of reference points. MHA stands for Multi-Head Attention Operation, output This represents the features after deformable attention shared by the reference point. Indicates a category or location branch, Indicates a category branch, This indicates a branch to be located.

[0058] More preferably, the method and system for detecting small-sample PCBA components based on a visual fundamental model uses a multi-layer shared classification and localization detection head to predict bounding boxes, and performs joint post-processing on the multi-layer results output.

[0059] More preferably, the method and system for detecting small-sample PCBA components based on a visual basic model includes an actuator, which is used to execute the method and system for detecting small-sample PCBA components based on a visual basic model described above.

[0060] The present invention will now be described through a more specific embodiment: The method provided in this embodiment mainly includes three stages: offline pre-training, online fine-tuning, and online inference.

[0061] 1. Offline pre-training stage This phase aims to train the model on a base dataset (such as PASCAL VOC) so that it learns general object representations and detection capabilities.

[0062] S101: Data preparation and feature extraction.

[0063] Obtain large-scale public object detection datasets, such as PASCAL VOC or COCO, as the source domain dataset. The dataset is divided into mini-batches. For each input image in each batch... We use a frozen DINOv2 visual base model (such as ViT-S or ViT-B) as a feature extractor to extract its single-scale feature maps. The DINOv2 model's parameters are frozen during the pre-training stage, and its powerful feature representation capabilities provide the model with high-quality global semantic information.

[0064] S102: Feature extension.

[0065] Since the localization task of object detection requires richer, finer-grained local information, while DINOv2's single-scale features focus more on global classification information, the features need to be expanded.

[0066] (1) Localization Feature Expansion: The single-scale features output from DINOv2 are input into the Scale Expansion Encoder (SEE). For example... Figure 3 As shown in the diagram, Figure (a) illustrates the overall architecture of SEE, Figure (b) is a schematic diagram of the Convolutional Block (CBS) module, and Figure (c) is a schematic diagram of the Cross-Scale Feature Fusion (CSFF) module. The scale-expanding encoder first reconstructs and downsamples features through the convolutional block, forming a multi-layered feature pyramid. The CBS module consists of convolutions, batch normalization, and the SiLU activation function. Self-attention is applied to the top-level features to aggregate the global context. Subsequently, top-down and bottom-up feature fusion is performed through the Cross-Scale Feature Fusion (CSFF) module. This module utilizes convolutions with different receptive fields to process cascaded features in parallel and stabilizes training through residual connections. The final output is localization features containing multi-scale detailed information. .

[0067] (2) Classification feature expansion: In order to maintain the high-quality global information of the original DINOv2 features for classification, bilinear interpolation is directly used to adjust the single-scale features of DINOv2 to the same spatial size as the multi-scale feature map output by SEE (e.g., 1 / 16 of the original image), thus forming classification features. .

[0068] S103: Prompt generation and query sampling.

[0069] At the same time, the same input image The input is fed into the frozen localization base model FastSAM (i.e., PGFN). Its segmentation branch is removed, and only the detection branch is used to accelerate inference. After post-processing (including Non-Maximum Suppression (NMS) and Top-K Selection), a fixed number (e.g., 900) of pre-location boxes (cue boxes) and their corresponding center points (serving as location indices for query points) are generated. This process can be formally represented as:

[0070] in, It is the FastSAM model. It is the NMS threshold (e.g., 0.8). This represents the number of queries. It is a collection of tooltip boxes, each element containing a box. . It is the corresponding set of indexes for query point locations.

[0071] Then, using a feature-aligned query sampling method, based on the position index... From the perspective of multi-scale classification features and location features Sample the corresponding query feature vector and :

[0072] This ensures that localization and classification tasks use features from the same location in the image for decoding.

[0073] S104: Task decoupling, decoding, and prediction.

[0074] Classification features Location features and the query features obtained from sampling , and prompt box collection They are input together into the query decoder for task decoupling. For example... Figure 4 As shown, the task-decoupled query decoder is composed of The system consists of stacked decoding layers, each containing parallel localization and classification branches. The processing flow for each branch includes: (1) Learnable linear mapping and self-attention.

[0075] (2) Reference-point shared deformable attention (PSCA): such as Figure 4 (b) shows the query points. The corresponding prompt box Generated through a linear layer One (e.g., 4) reference points Query characteristics for locating and classifying branches. and These reference point coordinates are shared, but each is derived from... and Medium-sampled feature values and Then, each task performs multi-head attention computation. This mechanism maintains the necessary geometric information exchange between tasks while decoupling them. This process can be formally represented as:

[0076]

[0077]

[0078] in, Indicates each query point The corresponding prompt box, and This represents the linear layer weights and biases used to offset the reference box coordinates. and These represent the location indices corresponding to the localization and classification tasks, respectively. Query characteristics, and This represents the Sigmoid activation function and its inverse function. This indicates the prompt point after the offset. Represents the features of the input image. This represents the features obtained through resampling. Indicates grid sampling operation, Indicates the number of reference points. MHA stands for Multi-Head Attention Operation, output This represents the features after deformable attention shared by the reference point. Indicates a category or location branch, Indicates a category branch, This indicates a branch to be located.

[0079] (3) Feedforward Neural Network (MLP).

[0080] Features output by the deformable attention layer with shared reference points and Input a 2-layer MLP to obtain features for classification. and features used for positioning .

[0081] In other embodiments, preferably, the task-decoupled query decoder adopts a dual-branch structure, combining self-attention and reference point sharing variability attention to process the query and feature input of the classification and localization branches; the specific process is expressed as follows:

[0082]

[0083]

[0084]

[0085] in, For the input category or location branch query, Indicates a category or location branch, Indicates a category branch, Indicates the location of the branch; For learnable positional encoding, Representing classification features, Indicates location features, This indicates a tooltip; MHA stands for Multi-Head Attention Operation, and LN stands for Layer Normalization Operation. This represents the query features after attention processing; PSCA represents deformable attention shared by reference points. This represents the deformable attention processing result of the classification branch. This represents the deformable attention processing result of the localization branch. This represents the features output by the decoder used for classification. This represents the features output by the decoder used for localization. and These represent two-layer MLPs: the classification branch and the localization branch, respectively, with parameters not shared between them.

[0086] The output of each decoder layer is connected to a shared classification head (single-layer linear layer) and a shared regression head (3-layer MLP) for directly predicting class scores and bounding box offsets. Input a single linear layer and pass it through a Sigmoid activation layer to obtain a classification score. .Will Input a 3-layer MLP to obtain the offset of the active bounding box. Use this offset to correct the tooltip. The specific process is as follows:

[0087] in and Let Sigmoid be the activation function and its inverse function. The input prompt box for the decoder; the MLP is a 3-layer multilayer perceptron. Output the result for the positioning box.

[0088] S105: Loss Calculation and Parameter Update.

[0089] A one-to-many sample allocation strategy is adopted, matching each ground truth bounding box with multiple predicted bounding boxes to increase the number of positive samples. The overall loss function is then calculated. It is obtained by weighted summation of the losses from the outputs of each decoder layer. The loss function includes: (1) Classification loss: Variational Loss (VFL) is used. ).

[0090] (2) Regression loss: L1 loss is used ( ) and Generalized IoU (GIoU, Weighted sum of losses ,

[0091] Where λ1, λ2, and λ3 are the weight hyperparameters of the loss function.

[0092] During the pre-training phase, the parameters of DINOv2 and FastSAM are frozen. The parameters of the scale-expanding encoder and the task-decoupled query decoder are updated only through backpropagation to minimize... Use the AdamW optimizer to train until the preset number of iterations is reached.

[0093] 2. Online fine-tuning stage This phase aims to enable the pre-trained model to quickly adapt to specific industrial inspection tasks using a very small number of target domain (industrial data) samples.

[0094] S201: Data preparation.

[0095] Obtain a small-scale dataset of the target industrial scenario (For example, 1, 5, or 10 images per category). Also divided into small batches.

[0096] S202: Forward Propagation and Loss Calculation.

[0097] For each batch of images, perform the exact same forward propagation process as S101-S104 to obtain the prediction results. Simultaneously, calculate two losses: (1) Target detection loss Similar to S105, calculate the VFL, L1, and GIoU losses between the predicted bounding boxes and the ground truth bounding boxes output by the task-decoupled query decoder.

[0098] (2) Class-independent localization loss This loss is used to fine-tune the cue generator, i.e., the base model FastSAM. It computes the cue boxes predicted by FastSAM. The loss between the bounding box and the ground truth bounding box. This loss only cares about the position of the bounding box and not its category, hence it is called class-independent. The loss function includes the binary cross-entropy loss (BCE). ), Distribution Focal Loss (DFL, ) and Complete IoU loss (CIoU, )

[0099] in, , , These are the weight hyperparameters of the loss function.

[0100] S203: Model fine-tuning and parameter updates.

[0101] At this stage, the model parameters are selectively updated.

[0102] For the localization base model FastSAM, in order to adapt to the target domain without destroying its original strong generalization ability, a low-rank convolutional adapter and a semi-supervised distillation strategy are used for fine-tuning.

[0103] (1) Low-rank convolutional adapter (LRCA): such as Figure 2 As shown in the diagram, Split and Concat refer to the splitting and merging operations, respectively. A low-rank branch is added next to the convolutional layer in FastSAM's C2F module. This branch consists of two convolutional kernels. ( Convolution kernels reduce the dimensionality of the original input channels to [a smaller value]. )and (with the original convolution kernel) Equal in size, with the channel number restored, they are connected in series. Forward propagation can be represented as...

[0104] Freeze during training Only update and During reasoning, the following is used:

[0105] By merging convolution calculations, the computational load is not increased and the convolution result remains unchanged.

[0106] (2) Semi-supervised distillation: using a frozen, untuned FastSAM model Generate pseudo-labels for the current training images . Real labels With pseudo-tags The merged signal, after being filtered by NMS, is used as a monitoring signal for fine-tuning the PGFN. This helps the model retain existing knowledge while adapting to new domains. This process can be formally represented as:

[0107]

[0108] Where X is the input image, and T represents the post-processing process, including... Non-maximum suppression with threshold and Filter by score based on threshold.

[0109] The scale-expanded encoder and task-decoupled query decoder unfreeze parameters and perform full fine-tuning.

[0110] The total fine-tuning loss is: The algorithm updates the parameters of the low-rank convolutional module, the scale-expanding encoder, and the task-decoupled query decoder by optimizing the algorithm, minimizing... The number of fine-tuning iterations is far fewer than in the pre-training phase.

[0111] 3. Online Reasoning Stage The trained model is deployed to the PCBA production site for actual testing.

[0112] S301: Feature extraction and suggestion generation.

[0113] Image to be detected Input model. Multi-scale localization features are obtained sequentially through DINOv2 and scale-extended encoder. Classification features were obtained through DINOv2 and interpolation. Simultaneously, a prompt box is generated using the finely tuned FastSAM positioning model. and query index .

[0114] S302: Decoding and Prediction.

[0115] according to from and The sampled query features are input to a task-decoupled query decoder along with R. The decoder outputs the classification scores and bounding box coordinates for all query points in parallel.

[0116] S303: Post-processing.

[0117] The outputs of all layers of the task-decoupled query decoder are aggregated or the output of the highest layer is selected. Post-processing techniques such as non-maximum suppression (NMS) are applied to filter out the final high-confidence, non-overlapping predicted boxes as the detection results output, based on the classification score and the overlap of the bounding boxes.

[0118] The detection method provided in this embodiment was tested on a publicly available cross-domain small sample dataset, and the results are as follows: Figure 5 As shown. Figure 5 The first row contains the natural scene dataset VOC, the second row contains the underwater cross-domain dataset FISH, and the third row contains the industrial cross-domain dataset NEU-DET.

[0119] The testing method provided in this embodiment demonstrates the following testing results for PCBA components in a real small-batch PCB production line: Figure 6 As shown, even when using only one PCBA sample to create the dataset, the model still achieves extremely high positioning accuracy and recognition accuracy. This proves that the model can effectively solve the problem of scarce samples in cross-domain industrial scenarios, facilitating further processing and analysis by other downstream algorithms, ensuring the yield rate of the production line, and improving production efficiency and stability.

[0120] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting small-sample PCBA components based on a visual fundamental model, characterized in that, Includes the following steps: Offline pre-training: The PCBA image dataset with object detection boxes is divided into multiple batches. The basic model DINOv2 is used to extract single-scale features of each image in each batch. Then, the single-scale features are expanded into multi-scale localization features and classification features by using a scale expansion encoder and bilinear interpolation, respectively. The FastSAM model is used to obtain the pre-location bounding box of each image in each batch. Query point features and prompt boxes are obtained from multi-scale localization features and classification features respectively. The multi-scale localization features, classification features, query point features and prompt boxes are simultaneously input into the task decoupled query decoder to predict the classification result and regression result for each query point feature. Calculate the object detection loss between the classification and regression results and the ground truth box, set the total number of iterations and the learning rate during the training process, freeze the base models DINOv2 and FastSAM, adjust the parameters in the scale expansion encoder and task decoupling query decoder to reduce the class-independent localization loss and object detection loss function, and the task decoupling query decoder corresponding to the preset total number of training iterations is the optimal task decoupling query decoder. Online fine-tuning: The PCBA image dataset used for fine-tuning is divided into multiple batches; the class-independent localization loss and object detection loss of the model training are obtained using the same steps as offline pre-training; FastSAM is fine-tuned using the convolutional low-rank fine-tuning module, and the class-independent localization loss between the pre-location box and the ground truth box is calculated; the total number of iterations and the learning rate are set during the training process, and the parameters in the convolutional low-rank fine-tuning module, the scale-expanded encoder class-independent pre-location head, and the task decoupling query decoder are adjusted to reduce the class-independent localization loss and object detection loss. When the preset total number of training iterations is reached, the corresponding task decoupling query decoder is the optimal task decoupling query decoder. Online reasoning: The PCBA images to be processed are input into DINOv2 to extract single-scale features of each image in each batch. Then, the single-scale features are expanded into multi-scale localization features and classification features by using a scale-expanding encoder and bilinear interpolation, respectively. FastSAM is used to obtain the pre-location bounding box of each image in each batch. Query point features and prompt boxes are obtained from multi-scale localization features and classification features respectively. Multi-scale localization features, classification features, query point features and prompt boxes are simultaneously input into the task decoupled query decoder to predict the classification result and regression result for each query point feature. The high-scoring non-overlapping prediction boxes are selected as the target detection result. The method utilizes a scale-expanding encoder and bilinear interpolation to expand single-scale features into multi-scale localization and classification features. The scale-expanding encoder aggregates top-level features using self-attention, followed by a dual-path fusion from top to bottom and bottom to top using a multi-receptive-field fusion module. The multi-receptive-field fusion module extracts features using a three-layer parallel multi-receptive-field structure and adds a residual structure to prevent gradient vanishing. The three-layer parallel multi-receptive-field structure includes a 1×1 convolution, a 3×3 convolution, and a cross-layer connection to process and sum the features.

2. The method for detecting small-sample PCBA components based on a visual fundamental model as described in claim 1, characterized in that, Bilinear interpolation was used to align the single-layer features extracted by DINOv2 with the original image in terms of size.

3. The method for detecting small-sample PCBA components based on a visual fundamental model as described in claim 1, characterized in that, The FastSAM base model is fine-tuned using a low-rank convolution module; the low-rank convolution calculation process is represented as follows: Where x represents the input feature, and Denotes a low-rank convolutional layer, where To transform features from the original number of channels Reduce to 1×1 convolution, To restore the feature channels as A convolution with the same kernel size as the original convolution. The original convolution kernel, For bias, This represents the convolution operation. During the inference phase, the associative law of convolution is used to merge the low-rank convolution kernel with the original convolution kernel, ensuring that the model structure remains unchanged before and after fine-tuning. This process is represented as: in, The merged convolution kernel is compared with the original convolution kernel. They are the same size.

4. The method for detecting small-sample PCBA components based on a visual fundamental model as described in claim 1, characterized in that, The FastSAM base model parameters are optimized through a self-supervised approach. This process first uses the untuned FastSAM model to predict targets on the image as pseudo-labels. Then, the ground truth values ​​and pseudo-labels are merged to serve as the actual labels for fine-tuning training, thus maintaining the performance of the original base model. The specific process is as follows: in, For the input image, This is the untuned FastSAM model. Indicates the post-processing procedure, including... Non-maximum suppression with threshold and Filter by score based on threshold. The pseudo-labels consist of high-confidence, non-repeating prediction boxes. This represents the training annotation set that combines ground truth and pseudo-labels.

5. The method for detecting small-sample PCBA components based on a visual fundamental model as described in claim 1, characterized in that, A feature-aligned query sampling method is used to obtain reference location indexes, cue reference boxes, and query features from classification and localization multi-scale features, respectively. The location index, tooltip reference box, and query features for classification and localization tasks have a one-to-one correspondence. A multi-layered shared classification and localization detection head is used to predict bounding boxes, and the multi-layered results are jointly post-processed.

6. The method for detecting small-sample PCBA components based on a visual fundamental model as described in claim 5, characterized in that, The reference location index and the prompt reference box are obtained through a query filtering mechanism, and the query features at the reference location index are obtained by aligning multi-scale feature maps. The specific process is as follows: in, For the input image, This indicates the multi-scale characteristics of flattening. This indicates the index of the query location on the feature map. The set, This represents the set of tooltip reference boxes corresponding to each location index; This indicates the corresponding feature found from multi-scale features based on the location index.

7. The method for detecting small-sample PCBA components based on a visual fundamental model as described in claim 1, characterized in that, The classification and regression processes are computed in parallel using a task-decoupled query decoder. The decoder takes as input the query feature set, reference box set, and multi-scale features for each of the localization and classification tasks, and outputs the classification and bounding box regression results for each query index.

8. The method for detecting small-sample PCBA components based on a visual fundamental model as described in claim 7, characterized in that, Deformable attention with reference point sharing is used to maintain some interactions between tasks; the specific process is as follows: in, Indicates each query point The corresponding prompt box, and This represents the linear layer weights and biases used to offset the reference box coordinates. and These represent the location indices corresponding to the localization and classification tasks, respectively. Query characteristics, and This represents the Sigmoid activation function and its inverse function. This indicates the prompt point after the offset. Represents the features of the input image. This represents the features obtained through resampling. Indicates grid sampling operation, Indicates the number of reference points. MHA stands for Multi-Head Attention Operation, output This represents the features after deformable attention shared by the reference point. Indicates a category or location branch, Indicates a category branch, This indicates a branch to be located.

9. A small-sample PCBA component target detection system based on a visual fundamental model, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the small sample PCBA component target detection method based on a visual fundamental model as described in any one of claims 1-8.

Images