A multi-modal super-resolution guided PCB defect detection method

Abstract

The application discloses a kind of multi-modal super-resolution guided PCB defect detection methods, belong to PCB defect detection technical field.The method is reconstructed to low-resolution image by super-resolution module, and the module successively executes: the multi-modal preprocessing of depth map is generated using DepthAnything model, the learnable degradation modeling of dynamic simulation degradation process is passed through neural network, the total variation reconstruction of denoising is carried out by fusing depth information, and the super-resolution generation based on R3GAN;Then, the high-resolution image is input into the improved RF-DETR detection network, and the number of refining times and feature layer selection are adaptively adjusted according to the defect size by dynamic recursive refiner (DRC), which significantly improves the detection accuracy and efficiency of PCB micro-defects.

Description

Technical Field

[0001] This invention relates to the field of PCB defect detection technology, and in particular to a multimodal super-resolution guided PCB defect detection method. Background Technology

[0002] As a core component of electronic products, PCBs are widely used in the manufacturing of electronic products such as computers, mobile phones, and televisions. To ensure PCB quality and prevent it from directly affecting product reliability and performance, bare board defect detection is essential. Common PCB defects include four types: burrs, notches, short circuits, and open circuits.

[0003] However, in actual production activities, the acquisition of PCB bare board defect datasets is affected by various factors during the acquisition process. These factors include hardware limitations such as mechanical vibration in the production environment, inaccurate focusing of the optical system, and sensor noise interference, as well as natural factors such as dust pollution and changes in ambient temperature and humidity. Consequently, the acquired defect images often suffer from varying degrees of blurring and insufficient resolution. This low-quality image data severely impacts the performance of subsequent defect detection algorithms, leading to increased false positive and false negative rates.

[0004] Some minute defects (such as poor solder joints, broken wires, micro-short circuits, pinholes, and copper residue) often have very few pixels, making them difficult to identify effectively using traditional detection methods, resulting in a high false negative rate. Although these minute defects are extremely small, they can cause open circuits, short circuits, or signal interference, seriously affecting product reliability. Summary of the Invention

[0005] In view of the problems existing in the prior art, the present invention is proposed.

[0006] Therefore, the problem to be solved by this invention is how to significantly reduce the missed detection rate of PCB micro-defects and overcome the low accuracy and poor real-time performance of micro-defect identification caused by insufficient image resolution, degradation distortion and rigid detection algorithms.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal super-resolution guided PCB defect detection method, comprising: acquiring a low-resolution color image of the PCB to be inspected; inputting the color image into a super-resolution module for reconstruction, outputting a high-resolution image; inputting the high-resolution image into a defect detection module, wherein the defect detection module employs an improved RF-DETR network, and its decoder integrates a dynamic recursive refinement controller (DRC) that adaptively adjusts the refinement times and feature layer selection based on the predicted defect size; the super-resolution module sequentially performs: multimodal data preprocessing to generate a depth map corresponding to the color image; learnable degradation modeling, dynamically generating a blur kernel and performing downsampling operations through a neural network; depth map-guided total variational reconstruction, fusing the depth map and color image information for denoising optimization; and super-resolution generation based on an R3GAN network.

[0008] In a preferred embodiment of the PCB defect detection method of the present invention, the depth map is generated by the DepthAnything model.

[0009] As a preferred embodiment of the PCB defect detection method of the present invention, the learnable degradation modeling includes a fuzzy kernel generation network: inputting a Gaussian distributed random variable, outputting a normalized fuzzy kernel through a convolutional layer; and a learnable downsampling network: inputting the image processed by the fuzzy kernel and the Gaussian distributed random variable, and achieving downsampling through a convolutional layer with a stride of 2.

[0010] As a preferred embodiment of the PCB defect detection method of the present invention, the loss function of the total variational reconstruction includes a data fidelity term, which constrains the pixel difference between the reconstructed image and the input image; and a regularization term, which fuses the gradient information of the depth map and the color map through bilateral filtering weights.

[0011] As a preferred embodiment of the PCB defect detection method of the present invention, the R3GAN network adopts a multi-scale receptive field refinement and progressive feature aggregation mechanism, and is trained by combining pixel loss, perceptual loss and adversarial loss.

[0012] As a preferred embodiment of the PCB defect detection method of the present invention, the Dynamic Recursive Refining Controller (DRC) executes the following strategy: when the defect size S < At that time, use 5 refinements and select feature layers P2-P4; when ≤S< When S ≥ 1, perform 3 refinements and select feature layers P3-P5; At that time, two refinements were performed and the P5 feature layer was selected; among them , The learnable thresholds are initially set to 16 pixels and 64 pixels, respectively.

[0013] As a preferred embodiment of the PCB defect detection method of the present invention, the Dynamic Recursive Refinement Controller (DRC) outputs continuous predicted values ​​during the training phase to ensure gradient backpropagation, and outputs discrete refinement instructions during the inference phase.

[0014] As a preferred embodiment of the PCB defect detection method of the present invention, the training of the defect detection module includes the following stages: a first stage: freezing the RF-DETR backbone network, training the DRC module and the detection head, and the loss function includes size-weighted regression loss; a second stage: unfreezing the backbone network and gradually increasing the proportion of small defect samples using a course learning strategy.

[0015] The beneficial effects of this invention are as follows: This invention significantly improves the identification capability of minute defects in PCBs through multimodal depth map-guided super-resolution reconstruction and a dynamic detection mechanism that adapts to defect size. In the super-resolution stage, the structural information of the depth map is fused with a learnable degradation model to effectively recover the texture features of minute defects lost due to image blurring and insufficient resolution, overcoming the limitations of traditional single RGB input in representing the three-dimensional structure of the circuit board. Simultaneously, the dynamic degradation network generates blurring and downsampling effects in real-world scenarios using random variables, greatly enhancing the adaptability of the super-resolution model to complex degradation environments on production lines. In the defect detection stage, the dynamic recursive refining controller automatically allocates refining times and feature layer resources based on defect size, achieving high sensitivity to minute defects and rapid response to large defects. This avoids the risk of missed detection of small targets by traditional uniform detection and reduces computational redundancy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the overall structure of the present invention.

[0018] Figure 2 This is a logic diagram of the super-resolution module of the present invention.

[0019] Figure 3 This is a schematic diagram of the learnable degradation modeling of the present invention.

[0020] Figure 4 This is a logic diagram of the defect detection module of the present invention.

[0021] Figure 5 This is a data flow diagram during the training process and in practical applications of this invention. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it an embodiment that is mutually exclusive, either alone or selectively, with other embodiments.

[0025] Reference Figures 1-5 This embodiment provides a multimodal super-resolution guided PCB defect detection method, including acquiring a low-resolution color image of the PCB to be detected;

[0026] The color image is input into the super-resolution module for reconstruction and outputs a high-resolution image. The super-resolution module is used to achieve image super-resolution, thereby obtaining a higher-pixel color image as the final super-resolution output, which is then input into the defect detection module to complete the final detection.

[0027] A high-resolution image is input into the defect detection module to obtain the defect detection results of the PCB. The detection results include: defect type, defect location, and defect confidence level.

[0028] The defect detection module uses an improved RF-DETR network, and its decoder integrates a dynamic recursive refining controller (DRC) that adaptively adjusts the refining times and feature layer selection based on the predicted defect size.

[0029] Specifically, the super-resolution module performs the following steps in sequence: multimodal data preprocessing, generating a depth map corresponding to the color image through a depth estimation model to construct the dataset basic estimation model for the entire super-resolution network; learnable degradation modeling, dynamically generating blur kernels and downsampling operations through a neural network; depth map-guided total variational reconstruction, fusing the depth map and color image information for denoising optimization; and super-resolution generation based on the R3GAN network.

[0030] Degradation modeling describes the mathematical process by which a high-quality image (HR) deteriorates into a low-quality image (LR) due to physical processes or noise interference. It is the foundation for image restoration tasks such as super-resolution and denoising. The core model is as follows:

[0031]

[0032] in, For original high-definition images, It is a degraded image; Represents the blur kernel (such as Gaussian blur, motion blur). This indicates a downsampling operation. This is additive noise (such as Gaussian noise). This model describes the degradation process of a high-resolution image to a low-resolution image. Super-resolution can be seen as the inverse of this process, that is, reconstructing a high-resolution image from a low-resolution image. When constructing the training set for a super-resolution network, a common method is to construct LR and SR image data pairs from a high-resolution image through downsampling and blur preprocessing. This invention uses the blur kernel and downsampling as the network training parameters, rather than a fixed preprocessing step, thereby improving the super-resolution performance.

[0033] Total variational reconstruction is used to initially denoise and deblur the downsampled image, obtaining a relatively clear low-resolution color image, thereby accelerating the training speed of the subsequent super-resolution network. This module uses a depth image-guided color image reconstruction method, the core of which is to improve the reconstruction quality of the color image by fusing the structural information of the depth image;

[0034] The defect detection module used in this invention is used to detect PCB defects. It employs a network based on RF-DETR with minor modifications for the specific task of PCB defect detection. RF-DETR is a Transformer-based detection model, and its network structure mainly utilizes four technical principles: Transformer architecture: adopting the encoder-decoder structure of DETR and using a self-attention mechanism to capture global context; DINOv2 backbone network: based on a large-scale self-supervised pre-trained feature extractor to improve small-sample adaptability; single-scale optimization: compared to the multi-scale design of Deformable DETR, it simplifies the feature map processing flow and reduces computational overhead; latency optimization: using TensorRT acceleration and incorporating NMS processing time into the overall latency evaluation system. This invention addresses the difficulty in detecting small targets in PCB defect detection, such as micro short circuits and fine cracks. It fine-tunes the recursive refinement in RF-DETR, proposing a Dynamic Refinement Controller (DRC) deeply integrated into the RF-DETR decoder. This DRC dynamically adjusts the number of recursive refinements through target size adaptation, achieving intelligent optimization of the detection process.

[0035] Specifically, the depth map is generated using the DepthAnything model;

[0036] It is worth noting that the depth estimation model used in this application is based on the open-source project Depth-Anything V2 (version number: v2.0).

[0037] Specifically, the learnable degradation modeling includes a fuzzy kernel generation network: inputting a Gaussian distributed random variable, outputting a normalized fuzzy kernel through a convolutional layer; and a learnable downsampling network: inputting the image processed by the fuzzy kernel and the Gaussian distributed random variable, and achieving downsampling through a convolutional layer with a stride of 2.

[0038] Fuzzy Kernel Generator Network: Input random variable zk ~ N(0,1), generate fuzzy kernel through convolutional network. The output layer Softmax ensures that the sum of kernel elements is 1. Finally, a blur kernel of size H×W×K×K is obtained to simulate real-world blur, including camera distortion, etc.; Downsampling network: adopts a 2-layer convolutional downsampling architecture (3×3 kernel, stride 2), inputting a blurred image and random variables zs~N(0,1), and outputting a low-resolution image to replace traditional fixed downsampling.

[0039] The loss function is:

[0040] in Pixel-level L1 loss ensures that the generated low-resolution image is consistent with the real low-resolution image in overall brightness and color distribution. Compared to L2 loss, L1 is more robust to outliers (such as noise) and avoids over-smoothing. It is used to calculate the gradient difference between the generated LR and the real LR, and to constrain the generated LR to align with the real LR on high-frequency details such as edges and textures. It prevents the generated image from losing important structural information due to blurring or downsampling. It is used to balance pixel precision and detail retention.

[0041] Specifically, the loss function of the total variational reconstruction includes a data fidelity term, which constrains the pixel differences between the reconstructed image and the input image; and a regularization term, which fuses the gradient information of the depth map and the color image through bilateral filtering weights.

[0042] Total variational networks (TVAs) transform image reconstruction into an optimization problem, using the structural information of high-resolution color images to guide iterative reconstruction of low-resolution depth maps. The core of this approach is to extract spatial and range weights from the color image through bilateral filtering, combining them with a total variational regularization term to suppress noise in the depth map while preserving edges. Finally, efficient optimization is achieved through the weight-sharing mechanism of a convolutional neural network, improving the resolution and structural accuracy of the depth map. Firstly, the image reconstruction network problem is modeled as a weighted sum of a data fidelity term and a total variational regularization term. The modeling in this invention is as follows:

[0043]

[0044] The previous item This indicates the similarity between the reconstructed image and the low-resolution image, the latter term. Used to solve the overfitting problem. The color image and depth image are obtained through bilateral filtering, respectively. Bilateral filtering involves the product of their respective spatial kernels and range kernels, thus achieving the fusion of the color image and depth image, rather than a simple fusion. M is the number of regularizers. It is a transformation matrix, which can be considered as a convolution operator for reconstructing the image, where It is a potential function. In this model, and These can be learned from network training, so none of them require additional computation. They are then improved using a simple gradient descent scheme, resulting in:

[0045]

[0046] Some parts use convolutional networks to automatically learn parameters through the network.

[0047] This invention is based on It is used as a loss function to combine constraints.

[0048] It is worth noting that, It uses HR color image results to guide LR depth image restoration, resolving the problem of blurred depth discontinuities. Depth weights. It is obtained by updating the depth image in each iteration. The quality of the later updated depth image is significantly better than that of the bicubic interpolated depth image, thus solving the texture duplication problem. ω and The combination of these factors will help the reconstruction model suppress the texture duplication problem in the image while maintaining a clearer depth discontinuity.

[0049] Specifically, the R3GAN network employs a multi-scale receptive field refinement and progressive feature aggregation mechanism, and is trained by combining pixel loss, perceptual loss, and adversarial loss.

[0050] It is worth noting that R3GAN is a generative adversarial network used for image super-resolution; GAN network super-resolution is a technique that applies generative adversarial networks to image super-resolution. The core logic is as follows: Generator: maps low-resolution (LR) images to high-resolution (HR) images, usually containing residual blocks and upsampling layers, focusing on restoring details; Discriminator: judges the authenticity of the generated HR image and the real HR image, outputs probability values, and guides the generator to optimize.

[0051] The core improvements of R3GAN are: multi-scale receptive field refinement: multiple convolutional kernels are used to extract local details and global features in parallel, and channel and spatial attention are used to fuse them to accurately enhance key regions; progressive feature aggregation: staged upsampling, low-frequency information is retained through skip connections, and intermediate supervision optimizes feature learning at each scale to avoid loss of details in high-magnification over-resolution; improved adversarial training: multi-scale discriminators evaluate generated images from different resolutions, and combine pixel, perceptual, adversarial and feature matching losses to balance detail generation and global structure and improve visual realism.

[0052] This invention combines the above three networks for joint training, and uses the trained fully variational network and R3GAN as the super-resolution module of this invention to process the input images, laying the foundation for subsequent defect detection.

[0053] Specifically, the Dynamic Recursive Refining Controller (DRC) executes the following strategy: when the defect size S < At that time, use 5 refinements and select feature layers P2-P4; when ≤S< When S ≥ 1, perform 3 refinements and select feature layers P3-P5; At that time, two refinements were performed and the P5 feature layer was selected; among them , The learnable thresholds were initially set to 16 pixels and 64 pixels, respectively.

[0054] This module first receives the initial prediction results from the RF-DETR decoder, extracts the physical size features of each predicted bounding box, and then uses trainable dual threshold parameters. and (Initial values ​​are 16px and 64px respectively. These initial values ​​are thresholds manually set based on the length of different defects on the PCB, serving as the dividing points between large, medium, and small defects. Small defects are defined as defects with a size S occupying less than 5% of the image width, medium defects occupying 5%–15% of the image width, and large defects occupying more than 15% of the image width. These values ​​can be manually adjusted based on the dataset.) The target is divided into three refined intervals: for tiny targets (S < ) Activate 5 enhancement refinements and activate P2-P4 high-resolution features, medium target ( ≤S< The process employs three standard refining steps using the P3-P5 balance characteristics, with a large target (S≥) Then perform two fast refinements to retain only the P5 semantic features.

[0055] Specifically, the Dynamic Recursive Refinement Controller (DRC) outputs continuous predicted values ​​during the training phase to ensure gradient backpropagation, and outputs discrete refinement instructions during the inference phase.

[0056] During training, the DRC module first calculates the expected value of the continuous refinement times (e.g., a real number between 2.0 and 5.0) and the soft weights of the feature layers at each scale based on the physical size of each predicted box. These values ​​participate in the recursive refinement calculation of the decoder in a weighted manner, enabling the network to perform end-to-end optimization of the shape of the refinement time curve and the threshold position during backpropagation. In the inference phase, the continuous output obtained from training is mapped to discrete decisions: firstly, the learned... , The targets are categorized into three types: small, medium, and large. A fixed number of refinement iterations (e.g., 5 / 3 / 2) are then output for each category, activating the corresponding feature layer combinations to form a clear recursive refinement instruction. This design enables the network to autonomously implement an intelligent detection strategy of "deep processing of small targets and fast passing of large targets." Through dynamic selection of multi-scale features and precise allocation of refinement iterations, it improves overall computational efficiency while ensuring the accuracy of detecting minute defects.

[0057] Specifically, the training of the defect detection module includes the following stages: first stage: freezing the RF-DETR backbone network, training the DRC module and the detection head, with the loss function including size-weighted regression loss; second stage: unfreezing the backbone network, and gradually increasing the proportion of small defect samples using a course learning strategy.

[0058] In RF-DETR, a Dynamic Recursive Refinement Control (DRC) module is introduced. During network initialization, the pre-trained weights of the RF-DETR backbone network are maintained, and the threshold values ​​are initialized in the DRC module parameters. / Gaussian distribution initialization is used, and the refinement prediction layer is initialized using Xavier and a two-stage fine-tuning strategy is employed. In the first stage, the backbone is frozen, and only the DRC module and the detector head are trained, using a size-weighted loss function.

[0059]

[0060] Classification loss Optimize the prediction accuracy of defect categories. Regression loss. Improving defect location accuracy (GIoULoss) and size-aware weights:

[0061]

[0062] ( (For defect area, smaller targets receive greater weight)

[0063] Threshold regularization term Constraint learnable threshold / The offset range. And α, β, γ are dynamically adjusted according to the defect size. In the second stage, the backbone network is unfrozen, and a course learning strategy is adopted to first focus on medium-sized defects and gradually add small defect samples. The refinement effect is monitored, and the process stops when the accuracy of the validation set is saturated. This fine-tuning method enables the DRC module to quickly adapt to the size distribution characteristics of PCB defects while maintaining the feature extraction capability of the backbone network.

[0064] During training, the three loss weight parameters α, β, and γ are dynamically adjusted according to the target size to achieve differentiated optimization for defects of different scales. First, the physical dimensions (width w and height h) of the predicted bounding box are normalized to obtain the size indices. And set a threshold for judging small targets. Then, a differentiable sigmoid function is used. Calculate the weight coefficient of the small target , where k controls the steepness of the curve. When the target is much smaller than the threshold, this weight is close to 1; when the target is much larger than the threshold, the weight is close to 0, thus achieving a smooth, high-weight allocation for small targets. Based on this coefficient, the adjustment rules for α, β, and γ are as follows:

[0065]

[0066] , , The base values ​​are set to 1.0, 2.0, and 0.1 respectively. , , The gain coefficients for small targets are set to 1.5, 3.0, and 2.0 respectively. These values ​​are fine-tuned during the training process. During training, when the defect size is very small... Approaching 1, , , Simultaneously, it is significantly amplified to improve classification accuracy, positioning accuracy, and the constraint strength of size / shape regularization; when the defect size approaches or exceeds the medium threshold, Gradually decreasing, , , The gradient converges to a base value, avoiding overweighting of large targets. This continuously differentiable dynamic adjustment mechanism ensures stable gradient propagation during training and can be converted to a fixed weight pattern during inference, enabling adaptive optimization for defects at different scales.

[0067] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-modal super-resolution guided PCB defect detection method, characterized in that: The application relates to a defect detection method and device for printed circuit boards (PCBs). A low-resolution color image of a PCB to be detected is acquired; The color image is input into a super-resolution module for reconstruction, and a high-resolution image is output; The high-resolution image is input into a defect detection module, and the defect detection module adopts an improved RF-DETR network, wherein a decoder of the RF-DETR network is integrated with a dynamic recursive refining controller (DRC) to adaptively adjust the refining times and feature layer selection according to the predicted defect size. The super-resolution module sequentially performs the following operations: multi-modal data preprocessing, generation of a depth map corresponding to the color image; learnable degradation modeling, dynamic generation of a blur kernel and down-sampling operation through a neural network; depth map guided total variation reconstruction, fusion of the depth map and color image information for denoising optimization; and R3GAN network based super-resolution generation. The dynamic recursive refinement controller (DRC) implements the following strategy: when defect size S < 0.5 um, enable 5 refinements and select P2-P4 feature layers. when defect size S < 0.5 um, enable 5 refinements and select P2-P4 feature layers. When ≤S< 3 times refining is enabled and P3-P5 feature layers are selected; when S≥ When S < 0.5, enable 2-pass refining and select P5 feature layer; wherein , are learnable thresholds, with initial values of 16 pixels and 64 pixels, respectively.

2. The PCB defect inspection method of claim 1, wherein: The depth map is generated through a DepthAnything model.

3. The PCB defect inspection method of claim 2, wherein: The learnable degradation modeling comprises a blur kernel generation network: input of a Gaussian distribution random variable, convolution layer output of a normalized blur kernel; and a learnable down-sampling network: input of an image processed by the blur kernel and a Gaussian distribution random variable, down-sampling through a convolution layer with a step of 2. The loss function of the total variation reconstruction comprises a data fidelity term for constraining the pixel difference between the reconstructed image and the input image; and a regularization term for fusing the gradient information of the depth map and the color image through a bilateral filtering weight.

4. The PCB defect inspection method of claim 3, wherein: The R3GAN network adopts a multi-scale receptive field refining and progressive feature aggregation mechanism, and is trained in combination with a pixel loss, a perception loss and an adversarial loss. The dynamic recursive refining controller (DRC) outputs continuous prediction values in the training stage to guarantee gradient backpropagation, and outputs discrete refining instructions in the inference stage.

5. The PCB defect inspection method of claim 4, wherein: The training of the defect detection module comprises a first stage: freezing of an RF-DETR backbone network, training of a dynamic recursive refining controller (DRC) and a detection head, and a loss function comprising a size weighted regression loss; and a second stage: unfreezing of the backbone network, and gradual increase of the proportion of micro-defect samples by adopting a curriculum learning strategy.

6. The PCB defect inspection method of claim 5, wherein: ​ 7. The PCB defect inspection method of claim 6, wherein: ​

Images