A circuit board defect detection method and system integrated into a cascade heterogeneous network

Abstract

The application discloses a kind of circuit board defect detection method and system integrated into cascade heterogeneous network, the circuit board defect detection method constructs defect detection model and carries out defect detection to the measured circuit board;The defect detection model is divided into three feature extraction stages, and the PCB defect features from local to global, from structure to semantics are processed in turn.The first stage is shallow feature extraction module, based on depth separable convolution and two-way cross collaborative attention focuses on extracting the local details and texture features of PCB image.The second stage is middle layer feature extraction module, adopts graph neural network structure, initializes and correlation screening feature nodes by dynamic node feature modeling and light graph attention network, and constructs the extraction of middle layer feature map assisted by graph structure.The third stage is deep feature extraction module, through the stacked state space enhanced self-attention encoder, and the entropy modulation quantum residual block across the coding layer is introduced to stabilize the training process.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a method and system for detecting circuit board defects by incorporating a cascaded heterogeneous network. Background Technology

[0002] Printed Circuit Boards (PCBs) are core components of electronic products, playing a crucial role in consumer electronics, communications, automotive electronics, medical devices, aerospace, and other fields. As the information-carrying platform of electronic systems, PCBs simultaneously provide electrical connections and mechanical support. In recent years, with the miniaturization, high density, and high reliability of electronic products, PCB manufacturing processes have become more complex: the number of layers has increased from the traditional 2-4 layers to 8-32 layers or even more, line width / spacing has shrunk to below 50 µm, and packaging devices have evolved from DIP types to BGA, QFN, and CSP, with micro-components represented by 01005 (0.4 × 0.2 mm) becoming increasingly common.

[0003] In this high-precision manufacturing environment, even the smallest defects, such as open circuits, short circuits, bridging, poor soldering, component misalignment, or incorrect polarity, can lead to functional failure or even serious safety accidents. For example, PCB defects in automotive electronics may affect the braking system; PCB problems in medical monitoring equipment may cause false alarms; and minute defects in PCBs in aerospace systems may cause catastrophic failures in extreme environments. Therefore, efficient and accurate PCB defect detection is a core element in ensuring the reliability and safety of electronic products.

[0004] PCB defect detection should be completed early in the manufacturing process to avoid significantly increased rework or scrap costs later in the assembly and testing stages, which could even lead to a batch recall in severe cases. Therefore, PCB quality inspection is directly related to product yield and first-pass yield (FPY), and affects production costs and corporate reputation.

[0005] In the past, inspection mainly relied on manual visual inspection, which was time-consuming, inefficient, tiring, and inconsistent. Therefore, automated inspection technology has become mainstream in the electronics manufacturing industry, including: 1) Automated Optical Inspection (AOI): using high-speed cameras and image processing to detect abnormal solder joints, bridging, missing solder joints, component misalignment, etc. on the PCB surface; 2) Automated X-ray Inspection (AXI): used to identify invisible solder joints, such as defects in the bottom pads of BGA components; 3) Automated Acoustic Inspection: used to identify structural defects such as internal delamination and layering.

[0006] AI and machine vision technologies have driven the widespread deployment of AOI systems, but traditional algorithms (such as template matching, threshold segmentation, and edge detection) are sensitive to noise and registration errors, leading to frequent false alarms and missed detections. In an industrial environment, excessively high false alarm rates increase the cost of manual review and reduce production line efficiency; missed detections may result in defective products being released, posing risks to customers and the brand.

[0007] In recent years, deep learning methods have been introduced into PCB defect detection. Some open-source PCB datasets, such as the DeepPCB dataset, have provided standardized benchmarks for the academic community, promoting the application of detection models such as YOLO, Faster R-CNN, and SSD in PCB defect detection. Compared to traditional methods, CNNs can automatically extract features, significantly improving detection accuracy. However, methods relying solely on CNNs have the following shortcomings: 1) Limited receptive field: CNNs focus on local textures and are insufficient in modeling global structures or long-range dependencies; 2) Lack of structural relationship modeling: Complex topological relationships exist between PCB components, which CNNs struggle to explicitly represent; 3) Difficulty in detecting small targets: Defects in PCB images are extremely small in scale and have low contrast, making it easy for CNNs to miss detections or be affected by noise.

[0008] Therefore, Graph Neural Networks (GNNs) have been introduced to model the graph structure relationships of components or solder joints. LightGCN is efficient due to its simplified propagation mechanism and is suitable for handling the topological dependencies of PCBs. Meanwhile, the Transformer architecture has the advantage of handling global features and long-range dependencies. The VisionTransformer proposed in the image domain has verified that the Transformer also demonstrates certain superiority in handling image problems, and its variants (such as Swin and DeiT) have been tried for PCB defect detection tasks. Similarly, for the Mamba model, which handles global architecture and interdependencies, there are increasingly more variants in the image domain, such as VisionMamba and MambaVision. Compared to Mamba, which is based on a selective state-space model, the Transformer can capture more refined dependencies when processing shorter sequences. However, when processing long sequences, the attention weights of key parts are diluted by a large number of irrelevant tokens. Furthermore, in terms of time complexity, Mamba has O(n) while the Transformer has O(n²). Therefore, a reasonable integration of Mamba and Transformer for complementarity is a new trend.

[0009] Given the advantages of Transformer in capturing refined local features and the complementary characteristics of Mamba (Selective State Space Model) in long-sequence global modeling and linear computational complexity, this invention aims to address the limitations of existing CNN networks in complex PCB defect detection, which struggle to simultaneously consider both minute target features and global structural dependencies. To this end, this invention designs a PCB defect detection method integrating Dynamic Difference Graph Convolution and the MGTNet cascade heterogeneous network (Mobile-GCN-Transformer Net). This method significantly reduces the false negative and false positive rates of minute defects by addressing the problems of feature degradation and computational redundancy in deep networks, effectively reducing scrap and rework costs in the PCB production process. Simultaneously, thanks to the introduction of linear computational complexity, the algorithm's dependence on high-end computing hardware is reduced, enabling the high-precision detection model to be adapted to low-cost edge computing devices, significantly improving the inspection throughput and intelligence level of the production line. The fusion architecture proposed in this invention provides a new technological paradigm for industrial visual inspection, contributing to the intelligent transformation and upgrading of the electronics manufacturing industry from labor-intensive to technology-intensive. Summary of the Invention

[0010] The purpose of this invention is to provide a method and system for detecting circuit board defects by incorporating a cascaded heterogeneous network.

[0011] In a first aspect, the present invention provides a method for detecting defects in a circuit board incorporating a cascaded heterogeneous network, the method comprising:

[0012] A defect detection model is constructed; the defect detection model includes a shallow feature extraction module, a middle feature extraction module and a deep feature extraction module connected in sequence; the shallow feature extraction module, the middle feature extraction module and the deep feature extraction module process defect features from local to global and from structure to semantics in sequence.

[0013] The mid-layer feature extraction module includes a reshaping module, a dynamic node feature generation module, a dynamic difference graph attention module, and a post-processing module connected in sequence. The reshaping module is used to adjust the size of the shallow feature map. The dynamic node feature generation module is used to cluster the pixel-level features in the output feature map of the reshaping module into multiple nodes and construct node features and edges between nodes. The dynamic difference graph attention module is used to obtain the relative change information between nodes based on the edges between nodes and integrate the relative change information into the node's own features. The post-processing module is used to constrain the output feature map of the dynamic difference graph attention module to obtain the mid-layer feature map.

[0014] Acquire images of the circuit board under test and preprocess the images; input the images of the circuit board under test into the defect detection model for defect detection.

[0015] Preferably, in the shallow feature extraction module, the input feature map is processed by multiple cascaded depthwise separable convolutional blocks, and the processing results are fused with the input feature map processed by the convolutional layer to obtain a spatial local feature map; a dual-path cross-cooperative attention module is used to process the spatial local feature map, and the processing results are fused with the input feature map processed by the convolutional layer to obtain the shallow feature map output by the shallow feature extraction module.

[0016] Preferably, the depthwise separable convolutional block includes a first convolutional block and a second convolutional block connected in series, and the fusion result of the output feature map of the second convolutional block and the input feature map of the depthwise separable convolutional block is used as the output feature map of the depthwise separable convolutional block; the first convolutional block includes a depthwise convolutional layer, a batch normalization layer and an activation function connected in sequence; the second convolutional block includes a pointwise convolutional layer, a batch normalization layer and an activation function connected in sequence.

[0017] Preferably, in the dual-path cross-cooperative attention module, convolutional layers of different sizes are used to process the input feature map to obtain two feature response values. Global average pooling is then performed on the two feature response values, followed by feature concatenation to obtain a fused feature map. A multilayer perceptron is used to convert the fused feature map into importance weights, which are then split into two dynamic weights. Two multilayer perceptrons are used to process the two dynamic weights respectively, and the processing results are reshaped before feature extraction is performed on the input feature map to obtain two feature representations. The two feature representations are then fused to obtain a channel-level feature map. Average pooling and max pooling operations are applied to the channel-level feature map respectively, followed by fusion to obtain a feature descriptor. A depthwise separable convolution is used to extract a spatial attention map from the feature descriptor, and the two feature response values ​​are weighted and fused based on the spatial attention map to obtain a shallow feature map extracted by the shallow feature extraction module.

[0018] Preferably, in the dynamic node feature generation module, a neural network is used to obtain the soft attribution matrix from the shallow feature map to multiple supernodes; the feature pixels in the shallow feature map are aggregated based on the soft attribution matrix, and the node features and center coordinates of each node are obtained according to the aggregation result; the edges between nodes are constructed using the center coordinates of each node.

[0019] Preferably, the dynamic difference graph attention module includes multiple stacked difference graph attention layers; each difference graph attention layer updates the input node features based on the product of difference features and attention coefficients; the difference features are the differences between the features of different nodes; the attention coefficients are constructed based on the difference features and similarity between different nodes.

[0020] Preferably, in the post-processing module, normalization and activation functions are used to process the feature map output by the dynamic difference map attention module, and the processing result is fused with the node features output by the dynamic node feature generation module to obtain the mid-level feature map.

[0021] Preferably, in the deep feature extraction module, two cascaded state-space enhanced self-attention encoders are used to process the input feature map to obtain a first intermediate feature map; the result of the entropy-modulated quantum residual module processing of the intermediate feature map is fused with the first intermediate feature map, and the fused result is processed by the state-space enhanced self-attention encoder to obtain a second intermediate feature map; the output feature map of the first state-space enhanced self-attention encoder is fused with the result of the entropy-modulated quantum residual module processing of the second intermediate feature map to obtain a fused feature map; the fused feature map is processed by the detection head to obtain the output result of the defect detection model.

[0022] Preferably, in the state-space enhanced self-attention encoder, the input feature map is fused with the location embedding to obtain the feature map. Enhanced self-attention module for feature maps using state space The processing is performed, and the processing results are compared with the feature map. After fusion, a normalization operation is performed to obtain a fused feature map. The fused feature map is then processed by a feedforward neural network, and the processing result is fused with the fused feature map before a layer normalization operation is performed to obtain the output feature map of the state-space enhanced self-attention encoder.

[0023] Preferably, in the state-space enhanced self-attention module, the feature map The query matrix, key matrix, and value matrix are projected respectively. The value matrix is ​​processed through a state space model, and the processing result is weighted and fused with the value matrix to obtain the final value matrix. The output feature map of the state space enhanced self-attention module is obtained by calculating the attention weights of the query matrix, key matrix, and final value matrix.

[0024] Preferably, the weights in the weighted fusion are obtained based on the degree of influence of the information of other nodes on the current node, and the greater the degree of influence, the higher the proportion of the value matrix.

[0025] Preferably, the entropy-modulated quantum residual block constructs the information barrier penetration probability based on the difference between the input feature distribution and the target distribution, and uses the product of the information barrier penetration probability and the input feature map as the output feature map.

[0026] Secondly, the present invention provides a circuit board defect detection system integrated with a cascaded heterogeneous network, which is used to perform the circuit board defect detection method described above. The circuit board defect detection system includes an image acquisition module, a preprocessing module, a defect detection module, and a visualization module. The image acquisition module is used to acquire images of the circuit board under test. The preprocessing module is used to scale, fill, convert color space, normalize, and tensor arrange the images under test. The defect detection module is used to detect defects in the preprocessed images. The visualization module is used to overlay the detection boxes and category labels output by the defect detection module onto the images under test and display them on a display screen.

[0027] Thirdly, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the aforementioned circuit board defect detection method.

[0028] Fourthly, the present invention provides a readable storage medium storing a computer program; when executed by a processor, the computer program is used to implement the aforementioned circuit board defect detection method.

[0029] The beneficial effects of this invention are:

[0030] 1. This invention, through a cascaded heterogeneous design of shallow, medium and deep layers, enables the model to possess the local perception capability of convolution, the relational modeling capability of graph structure, and the global interaction capability of Transformer, making it suitable for complex visual tasks with multiple scales, multiple forms and strong structural correlations in PCB defect detection.

[0031] 2. This invention integrates depthwise separable convolution and dual-path cross-cooperative attention module when performing shallow feature extraction, which preserves the pixel-level features of tiny defects in printed circuit boards to the greatest extent and significantly reduces the missed detection rate of PCB circuit boards with micro-defects such as pinholes and mouse bites.

[0032] 3. This invention breaks through the limitation of the local receptive field of the shallow feature extraction module by adding a dynamic node feature generation module to the mid-layer feature extraction module. By explicitly modeling the connection relationship between PCB components through dynamic graph structure, it effectively improves the recognition accuracy of structural defects such as open circuits and short circuits.

[0033] 4. The state-space enhanced self-attention encoder designed in this invention combines the linear complexity of SSM with the global attention advantage of Transformer, which significantly reduces the consumption of computing resources while ensuring the ability to capture long-range dependencies, making it suitable for industrial-grade deployment. Attached Figure Description

[0034] Figure 1 This is a structural diagram of the defect detection model constructed in this invention.

[0035] Figure 2 This is a structural diagram of the shallow feature extraction module in this invention.

[0036] Figure 3 This is a diagram of the depth separation convolution block structure in this invention.

[0037] Figure 4 This is a structural diagram of the dual-path cross-cooperative attention module in this invention.

[0038] Figure 5 This is a structural diagram of the mid-layer feature extraction module in this invention.

[0039] Figure 6 This is a structural diagram of the deep feature extraction module in this invention.

[0040] Figure 7 This is a structural diagram of the state-space enhanced self-attention encoder in this invention.

[0041] Figure 8 This is a structural diagram of the state-space enhanced self-attention module in this invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings.

[0043] A circuit board defect detection method integrating a cascaded heterogeneous network is disclosed. The circuit board defect detection system includes an image acquisition module, a preprocessing module, a defect detection module, and a visualization module. The image acquisition module acquires images of the circuit board under test. The preprocessing module performs scaling, filling, color space conversion, normalization, and tensor arrangement on the image. The defect detection module detects defects in the preprocessed image. The visualization module overlays the detection bounding boxes and category labels output by the defect detection module onto the image and displays them on a screen.

[0044] The circuit board defect detection method includes the following steps:

[0045] Step 1: Building the dataset

[0046] A dataset was constructed by collecting circuit board images containing different defect types, and the dataset was labeled to indicate the defect category and defect area in each circuit board image.

[0047] Step 2: Construct a defect detection model

[0048] like Figure 1 As shown, the defect detection model includes a shallow feature extraction module, a mid-level feature extraction module, and a deep feature extraction module connected in sequence. The shallow feature extraction module is used to extract circuit board images. The local details and texture features are extracted; where H and W represent the spatial height and width of the circuit board image, respectively; and C represents the number of channels in the circuit board image. The mid-layer feature extraction module is used to extract mid-layer feature maps from the shallow-layer feature extraction module; the deep-layer feature extraction module is used to process the mid-layer feature maps to obtain the circuit board image detection results.

[0049] 2-1. Shallow Feature Extraction Module

[0050] like Figure 2 As shown, the shallow feature extraction module includes multiple cascaded depthwise separable convolutional blocks and a dual-path cross-cooperative attention module. In the shallow feature extraction module, the input feature map (circuit board image) is processed by multiple cascaded depthwise separable convolutional blocks, and the processing results are fused with the input feature map processed by a 1×1 convolutional layer to obtain a spatial local feature map. By processing the input feature map through stacked depthwise separable convolutional blocks, spatial local features can be extracted layer by layer while maintaining lightweight parameters. The above process is expressed by the following formula:

[0051]

[0052]

[0053] in, The result of processing the input feature map through a convolutional layer; This represents the convolution kernel used to align the output feature dimensions; Input feature map; Represents a local feature map of space; This represents the output feature map of multiple concatenated depthwise separable convolutional blocks.

[0054] The spatial local feature map is processed using a dual-path cross-cooperative attention module, which performs feature cross-cooperative enhancement in both the channel and spatial dimensions to highlight the local differences in the defective region. The processing results of the dual-path cross-cooperative attention module are then fused with the input feature map and processed by a 1×1 convolutional layer to obtain the shallow feature map output by the shallow feature extraction module.

[0055] 2-1-1. Depth-separable convolutional blocks

[0056] like Figure 3 As shown, the depthwise separable convolutional block employs a structure that separates depthwise convolution and pointwise convolution, reducing computational cost while enhancing the local receptive field. Within the depthwise separable convolutional block, the input feature map is sequentially processed through depthwise convolutional layers, batch normalization layers, and the NeLU activation function. Each channel is processed to eliminate internal covariate bias, accelerating model convergence and effectively mitigating the vanishing gradient problem. The processed results are then linearly combined using pointwise convolution, and the combined result is... The feature maps are fused with the input feature maps to obtain the output feature maps of depthwise separable convolutional blocks. Compared to standard convolution, depthwise separable convolution has a computational cost that is ( ) times that of standard convolution. The above process can be expressed mathematically as follows:

[0057]

[0058]

[0059]

[0060] in, This is the output corresponding to the c-th feature channel; The kernel size; This represents a depthwise convolution kernel applied independently to each channel; ; ; This represents the characteristic value of the m-th output channel at position (h, w); It is a convolution kernel for pointwise convolution.

[0061] 2-1-2. Dual-path cross-cooperative attention module

[0062] While convolutional neural networks excel at handling local defects, the extracted local features cannot cover the overall structural distribution of the circuit board and easily overlook key contextual information. To enrich the representational capabilities of features, dual-path cross-cooperative attention is used to achieve deep interaction and adaptive fusion of global and local feature information. By utilizing cascaded convolution operations, multi-scale pooling, and attention weighting strategies, the feature space is finely reconstructed.

[0063] like Figure 4 As shown, in the dual-path cross-cooperative attention, 3×3 convolutional layers and 5×5 convolutional layers are used to process the input feature map to capture multi-scale spatial information under different receptive fields and obtain feature response values. and characteristic response value It is represented as:

[0064]

[0065]

[0066] in, , K represents the feature response value of the output feature map of each branch at the k-th channel of spatial coordinates (i,j); (3) and K (5) These correspond to the weight kernel tensors for 3×3 and 5×5 convolution operations, respectively. The input feature map is used; the summation operation is represented in the kernel window (3×3 or 5×5) and the input channel. Perform a sliding dot product on top, and finally obtain , (H3=H5=H',W3=W5=W').

[0067] To obtain descriptors for the two convolution operations, global average pooling is performed on the two feature response values ​​to compress the spatial dimension (H×W) into a channel-level vector, capturing the global statistical information of each channel and providing input for the subsequent attention mechanism. The global pooling operation for the feature descriptor on the s-th channel is as follows:

[0068]

[0069]

[0070] in, and This is the output of the global average pooling operation; Indicates input features The feature at (i,j) on the s-th channel.

[0071] The two global statistical information sets are concatenated to obtain a fused feature map. It is represented as:

[0072]

[0073] Using a multilayer perceptron (MLP) to fuse feature maps The process involves fusing and comparing components within the same feature space while reducing the number of parameters. The multilayer perceptron (MLP) consists of a first fully connected layer, a ReLU activation function, a second fully connected layer, and a sigmoid activation function; importance weights normalized to the (0,1) range are obtained through MLP processing. It is represented as:

[0074]

[0075] in, Hyperparameters This represents the reduction ratio, which is usually 2 or 4. Based on experimental comparisons, this method sets r=2. Represents the sigmoid function; This represents the ReLU activation function.

[0076] Importance weights The data is segmented, with the information from the first D channels denoted as... It contains weights from the dual-path information that are biased towards guiding the 3×3 path; the information from the last D channels is denoted as... It contains weights that are biased towards guiding 5x5 paths, and the implementation process is as follows:

[0077]

[0078] in, For dynamic weights.

[0079] The convolutional kernels are adaptively adjusted based on the input, and two branches generate two convolutional kernels of different sizes. A 5×5 convolutional kernel is generated using DW3, which contains local detail information, ensuring that local sensitivity is not lost during its generation. Similarly, a 3×3 convolutional kernel is generated using DW5, which contains wide-area context information, enhancing its ability to capture contextual information. This allows the smaller convolutional kernel to be context-aware during local computation, improving feature representation. Both convolutional kernels are parameterized and generated using independent multilayer perceptrons, as follows:

[0080]

[0081] in, , , , , , , , , where represents the weights of two independent MLPs, d is the hidden layer size, and N is the number of output channels. Represents the simoid function. Represents the ReLU activation function. These represent the weights and biases, respectively. The outputs of BDK5 and BDK3 are reshaped to obtain dynamic convolutional kernels. and .

[0082] Using convolutional kernels with dynamic features to process the input feature map Performing dynamic convolution operations yields deeper, multi-scale feature representations. and This process performs channel-level feature extraction on the original input feature map, which is represented as follows:

[0083]

[0084] Where h and w represent the spatial indexes (height and width) of the output feature map, respectively, and n is the output channel index. To maintain the output spatial size, padding is 2 for 5×5 convolutions and 1 for 3×3 convolutions. h and w are the spatial indices of the output feature map and , n is the output channel index, i and j are the spatial indices within the convolution kernel, and c is the input channel index; since the padding is 2, the indices of the input F need to be offset by i−2 and j−2 to ensure alignment.

[0085] To perform multi-scale channel-level feature fusion, the feature representation is... and By performing addition and fusion, channel-level feature maps are obtained. This is used for subsequent extraction of spatial feature maps, significantly improving the model's adaptability to multi-scale targets. The advantage of additive fusion lies in its computational efficiency and preservation of the original feature intensity distribution, avoiding the parameter inflation problem that may be caused by channel splicing, thus enhancing feature representation while controlling computational complexity. It is expressed as:

[0086]

[0087] Spatial attention maps are generated by leveraging the spatial relationships between features. Channel attention focuses on the importance of features represented by different channels, while spatial attention focuses on the importance of different spatial locations within the feature map, enabling the localization of specific regions containing key objects in the image. After applying average pooling and max pooling operations along the channel axes, these are concatenated to generate effective feature descriptors. The process is represented as:

[0088]

[0089]

[0090]

[0091] To generate spatial attention maps and enhance feature representations, while appropriately reducing computational cost and increasing the receptive field, a depthwise separable convolution with a kernel size of 7 is used to extract the spatial attention map. The implementation method is as follows:

[0092]

[0093]

[0094]

[0095] in, It is the convolution kernel for channel c; pad is 3 to maintain the same spatial size; Pointwise convolution weights.

[0096] Using F w Dot product with convolutional feature maps at multiple scales guides the focus of attention in feature maps at different scales, resulting in the feature map. and feature map It is represented as:

[0097]

[0098] in. This represents the feature representation obtained after performing a 3×3 convolution on the input features. This represents the feature representation obtained after 5×5 convolution of the input features, which incorporates channel and spatial attention weights F based on dynamic adaptive programming. w The features obtained from 3×3 convolution and 5×5 convolution are It can capture the features in the image more comprehensively and accurately.

[0099] For feature maps and feature map The fusion is performed, and the fusion result is combined with the input features after a 1×1 convolution transformation. Perform residual connections to obtain the shallow feature map extracted by the shallow feature extraction module. It is represented as:

[0100]

[0101] 2-2. Mid-layer Feature Extraction Module

[0102] To fully utilize the topological relationship between PCB component solder holes and traces, this invention proposes using GCN (Graph Convolutional Neural Network) as an intermediate layer to process the entire circuit board topology and obtain a representation of the relationship between each component solder hole node and trace in the PCB.

[0103] like Figure 5 As shown, the mid-level feature extraction module includes a reshaping module, a dynamic node feature generation module, a dynamic difference map attention module, and a post-processing module connected in sequence. The reshaping module is used to adjust the size of the shallow feature map; the dynamic node feature generation module is used to cluster the pixel-level features in the output feature map of the reshaping module into multiple nodes and construct the node features and the edges between the nodes; the dynamic difference map attention module is used to fuse the node's own features with the relative change information between nodes; the post-processing module is used to constrain the dynamic difference map attention module to obtain the mid-level feature map.

[0104] 2-2-1. Remodeling Module

[0105] In the reshaping module, the shallow feature map is... Perform a reshaping operation to obtain the reshaped feature map. , which is represented as;

[0106]

[0107] 2-2-2. Dynamic Node Feature Generation Module

[0108] The graph nodes are constructed using a learnable soft clustering approach. This method clusters pixel-level features into supernodes through end-to-end learning and uses a small neural network with two cascaded convolutional layers to learn shallow feature maps. Soft home matrix to M supernodes The process is as follows:

[0109]

[0110]

[0111]

[0112] in, This is a local feature map; This represents a 3×3 convolution kernel used to extract local features; This is an intermediate feature map; This represents a 1×1 convolution kernel, used to adjust the number of output channels.

[0113] The soft attribution matrix S is interpreted as the probability distribution of each pixel with respect to M nodes. Therefore, guided by the soft attribution matrix, each feature pixel is aggregated according to its affinity with different nodes, becoming a node in the graph structure, thus obtaining a preliminary representation of the graph nodes. To improve training stability and convergence speed, and to prevent certain feature values ​​from dominating the model due to excessively large values, normalization is used for the initial representation. By applying restrictions, the final node features are obtained. The center coordinates of each node The pixels it aggregates A joint decision. The above process is represented as follows:

[0114]

[0115]

[0116]

[0117] Where μ and σ represent the mean and variance, respectively.

[0118] Utilizing node features and node coordinates Generate edges between adjacent nodes. Calculate the distance and similarity between different nodes and use these as the basic metric, represented as:

[0119]

[0120] in, The distance between different nodes; Spatial-semantic tradeoff coefficient

[0121] For each node i, select the k nearest neighbors of node i based on the distance D(i,j). It is represented as:

[0122]

[0123] To improve the stability of the neighborhood, a semi-reciprocal approach is used to construct edges between nodes. and edge set It is represented as:

[0124]

[0125] Based on node features Sum of edges Construct a topological graph representation G=(Vertex, E).

[0126] An adjacency matrix is ​​constructed based on the relationships between different nodes. Adjacency matrix elements in Represented as:

[0127]

[0128] To improve the stability of the dynamic difference graph attention module during training, the adjacency matrix is... Adding self-loop connections yields the final adjacency matrix. adjacency matrix elements in .

[0129] Using adjacency matrix Calculate the degree matrix .

[0130] 2-2-3. Dynamic Difference Graph Attention Module

[0131] In the dynamic difference graph attention module, the difference features between nodes in each difference graph attention layer are used to supplement attention. The absolute features of each node are fused with the relative change information between nodes, allowing the model to not only focus on "who is important" but also perceive "what are the differences between them." The difference features of node i and node j at layer l are obtained respectively. Attention coefficient The method to obtain it is as follows:

[0132]

[0133]

[0134] in, Represent the features of node i and node j in the l-th layer, respectively; Indicates the shared attention parameters; Represents the difference eigenvalue transformation matrix; Similarity feature transformation matrix; symbol Indicates a splicing operation; The similarity between node i and node j at layer l is obtained as follows:

[0135]

[0136] To prevent the model from becoming too large or causing gradient explosion during training and to improve model training stability, the Softmax function is used to normalize the attention coefficients between two nodes, thus obtaining the attention weights. It is represented as:

[0137]

[0138] The feature update formula for node i in the l-th layer of the dynamic difference graph attention module is as follows:

[0139]

[0140] in, The features of node i at layer l+1; These are learnable scalar coefficients used to control the overall contribution of the difference features to the update of the central node; The basic aggregation weight is obtained by normalizing the adjacency matrix, and the calculation process is as follows:

[0141]

[0142] After multi-layer aggregation, the output features of the dynamic difference map attention module are obtained. Represented as:

[0143]

[0144] in, This indicates the importance of the l-th layer embedding in the final embedding layer; This represents the number of attention layers in the difference graph.

[0145] In this embodiment, the dynamic difference graph attention module employs two stacked difference graph attention layers to achieve a balance between model performance and engineering realities: a single layer has a limited receptive field, making it difficult to capture complex global topological relationships; while an excessively deep network leads to an increase in the number of parameters and computational latency, which is detrimental to industrial deployment. The two-layer structure allows it to aggregate second-order neighbor information to fully understand the local topological structure while maintaining the model's lightweight nature, thus balancing learning performance with the industry's requirements for low latency and high-efficiency deployment.

[0146] 2-2-4. Post-processing module

[0147] In the post-processing module, the output features are... RMSNorm normalization and Tanh activation function are applied sequentially to stabilize training and constrain the output range to the [-1, 1] interval. For the feature vector of each node i... ( ), its final output Represented as:

[0148]

[0149] To prevent gradient vanishing and decreased training stability, and for dimension alignment, the processed output is... Compared with the original node features Perform residual connections to obtain the mid-layer feature map extracted by the mid-layer feature extraction module. It is represented as:

[0150]

[0151] Mid-layer feature map It integrates the topology of the entire PCB component solder holes and traces, providing richer features for subsequent PCB classification processing.

[0152] 2-3. Deep Feature Extraction Module

[0153] like Figure 6As shown, in the deep feature extraction module, two cascaded state-space enhanced self-attention encoders are used to process the input feature map (intermediate feature map) to obtain the first intermediate feature map; the result of the intermediate feature map processed by the entropy-modulated quantum residual module is fused with the first intermediate feature map, and the fused result is processed by the state-space enhanced self-attention encoder to obtain the second intermediate feature map; the output feature map of the first state-space enhanced self-attention encoder is fused with the result of the entropy-modulated quantum residual module and the second intermediate feature map to obtain the fused feature map; the detection head is used to process the fused feature map to obtain the output result of the defect detection model.

[0154] 2-3-1. State-space augmented self-attention encoder

[0155] like Figure 7 As shown, in the state-space augmented self-attention encoder, the input feature map (middle-layer feature map) is... The feature map is obtained by fusing the location embedding with the feature map. Feature map Represented as:

[0156]

[0157] Wherein, PE(C) represents the position code of the node center position C, which is obtained as follows:

[0158]

[0159] in, ; .

[0160] Enhanced self-attention module for feature maps using state space The processing is performed, and the processing results are compared with the feature map. After fusion, layer normalization is performed to obtain the fused feature map. ; Fusing feature maps through a feedforward neural network The process is performed, and the results are compared with the fused feature map. After fusion, layer normalization is performed to obtain the output feature map of the state-space enhanced self-attention encoder. The above process can be represented as:

[0161]

[0162] in, Presentation layer normalization operation; This represents a feedforward neural network.

[0163] The core advantage of self-attention mechanisms lies in their ability to capture global dependencies between any positions in a sequence. However, this mechanism has two significant limitations: first, its computational complexity increases quadratically with the sequence length L, O(L... 2 Firstly, the attention mechanism has limitations in long sequence scenarios. Secondly, when the sequence is too long, the attention weights may become excessively dispersed, making it difficult for the model to focus on key information. To effectively overcome these bottlenecks while retaining the local interaction capabilities of self-attention, a Z-row scanning state-space model (SSM) is designed to model the value vector (V) sequence. The core idea of ​​this hybrid architecture is to allow each position to not only obtain local, immediate contextual information through the attention mechanism when generating output, but also to integrate the long-term dependencies of the entire sequence through the state propagation of SSM.

[0164] like Figure 8 As shown, in the state-space enhanced self-attention module, the feature map is... Projected into query matrices respectively Key matrix Sum matrix Its projection method is as follows:

[0165]

[0166] in, represents the learnable weights; E represents the embedding space dimension.

[0167] Based on query matrix Bond matrix The degree of influence of information from other nodes on the current node is calculated. It is represented as:

[0168]

[0169] Where M is the number of nodes; and For query matrix Elements in; and Key matrix The elements in.

[0170] Using value matrix Projection vectors are generated in parallel using a multilayer perceptron. and selection vector It is represented as:

[0171]

[0172] in, This represents the k-th row of the value matrix V; and As weight; and P is the bias; P and r are the hidden layer sizes of the multilayer perceptron.

[0173] In the selective state space, construct the affine state update for each step to obtain the state matrix at step k. Represented as:

[0174]

[0175] in, The state space matrix; Let represent the bias matrix at step k.

[0176] Because high-dimensional state transition matrices are difficult to parameterize and their stability is hard to guarantee, a state space matrix is ​​used to simultaneously satisfy expressibility, stabilization, and parallelizability. and bias matrix The method to obtain it is as follows:

[0177]

[0178] in, It represents a low-rank decomposition basis.

[0179] Based on state space matrix and bias matrix Constructing affine tuples The transformation in this step is represented as follows:

[0180]

[0181] To define parallel affine combinations, we define combinations between asynchronous steps, where two steps... and The combination is:

[0182]

[0183] Transformation matrix computed in parallel during prefix scan Represented as:

[0184]

[0185] Final state Represented as:

[0186]

[0187] set up ,but When considering parallel merging, matrix multiplication... This would disrupt the low-rank structure, therefore the matrix portion is compressed after each merge, as shown below:

[0188]

[0189] in, It is a rank maintenance The compression operator is used. SVD rank-r truncation is employed, and the truncation formula is as follows:

[0190]

[0191] Through iterative calculation The final feature map of the state space output is obtained. .

[0192] To further enhance originality, a gating mechanism control value matrix V and a feature map are designed. The final value matrix is ​​obtained by the fusion ratio; gating By degree of impact This driving force ensures that the final feature representation simultaneously includes local details and global context, thereby improving the model's performance on the task. Final value matrix Represented as:

[0193]

[0194] in, For the final value matrix

[0195] Based on query matrix Key matrix Calculate attention weights from the final value matrix. It is represented as:

[0196]

[0197] It is beneficial to project queries, keys, and values ​​onto different dimensions using different learnable linear projections. Conversely, performing an attention function using only a single dimension of key, value, and query will inevitably result in less rich output compared to processing multiple dimensions together. Therefore, a multi-head processing strategy is advantageous, as the attention function is executed in parallel for each projection of the query, key, and value, producing multi-dimensional output. Then, each output is concatenated and projected again to obtain the output feature map of the state-space enhanced self-attention module. It is represented as:

[0198]

[0199]

[0200] in, Let represent the projection vector at the i-th head. This represents the combined projection matrix after splicing multiple heads. This represents the dimension of the hidden layer after specifying the number of heads, with a size of [value missing]. This leads to the multi-head attention representation of the entire input; h is the number of projection heads, h=4.

[0201] 2-3-2. Entropy-modulated quantum residual block

[0202] From an information transmission perspective, to alleviate the gradient vanishing problem and information decay effects during propagation between structural blocks, inspired by the concept of potential barriers in quantum physics, the greater the difference between the input feature distribution P(x) and the target distribution P(y|x) (where y is the task objective, such as a classification label), the higher the "difficulty" for the model to extract useful information. This difficulty in information extraction is defined as the information barrier. Combining information theory, information entropy is used to quantify the barrier strength; the greater the entropy, the higher the barrier. The information entropy of different nodes is then obtained. Its expression is:

[0203]

[0204] in, This is the weight matrix; For bias; Representation of spatial dimensions; This represents the normalized spatial dimension. ; This represents the number of tokens.

[0205] The middle layer feature maps are respectively The spatial dimension representation Z is mapped to a contrast space through an MLP to obtain projected features. and projection features ; respectively for projection features and projection features Perform a normalization operation to adjust the data to a specified range, resulting in normalized projected features. and projection features This prevents collapse and maintains the decoupling of the original feature information from the contrastive learning space; the mapping process is represented as:

[0206]

[0207] in, Indicates the weights during projection; Indicates the offset during projection.

[0208] Obtaining projection features and projection features similarity between It is represented as:

[0209]

[0210] in, and Projection features and projection features The elements in.

[0211] Since mutual information cannot be directly calculated, a lower bound approximation is introduced for contrastive learning. The essence of contrastive learning is to bring similar samples closer together and push dissimilar samples further apart, thereby learning the inherent structure and essential features of the data, based on similarity. Obtaining the InfoNCE loss from contrastive learning Its expression is:

[0212]

[0213] The positive samples have equal indices. The InfoNCE loss can be calculated using the above process, and then... Mutual information is obtained by calculating a lower bound with the number of tokens. Its expression is:

[0214]

[0215] The difference between mutual information and entropy is used to measure the strength of information penetration. Its expression is:

[0216]

[0217] in, This represents the learnable reference entropy, used to limit the strength of penetration; a larger value indicates a higher potential barrier. During model initialization, it is set to... It is given an empirical initial value (such as given based on the initial statistics of the input features), but is automatically optimized as a learnable parameter during backpropagation during training.

[0218] Based on information penetration strength Get penetration probability Its expression is:

[0219]

[0220] in, This represents the sigmoid function, used to map penetration intensity to a probability representation; It is a learnable gated scaling factor. Indicates shape as A single vector is used to expand the output to a channel-dimensional output.

[0221] Based on the penetration probability of each channel Constructing the probability of information barrier penetration This refers to the probability that the original feature will be passed to the next layer.

[0222] Using the probability of information barrier penetration The input feature map of the entropy-modulated quantum residual block is multiplied by the input feature map and used as the output feature map of the entropy-modulated quantum residual block.

[0223] 2-3-3. Detection head

[0224] The inspection head applies RMS regularization and the Nelu function to activate the input feature map. A Softmax classifier then transforms the activated features into a probability distribution, thereby classifying defects in different regions of the PCB image and outputting the classification results. Its expression is:

[0225]

[0226] Here, RMSNorm represents the root mean square regularization operation, which is used to improve the stability of features; the Nelu function introduces nonlinear transformations to enhance the expressive power of the model; This is the input feature map for the detection head.

[0227] Step 3: Training the model

[0228] The defect detection model is trained using a dataset. During training, based on the MGTNet cascade hybrid detection network architecture designed according to this invention, a multi-task, multi-scale joint optimization strategy is adopted to effectively address the problems of extreme imbalance between positive and negative samples, small defect scale, and severe background noise interference in PCB defect detection, and to fully utilize the topological features extracted by the second-layer dynamic difference graph network. The designed loss function... Including main classification loss and graph-assisted loss and contrastive learning loss It is represented as:

[0229]

[0230] in, and The classification loss and regression loss are the final outputs of the network; To illustrate auxiliary loss; The contrastive learning loss calculated for the deep feature extraction module is used as a regularization term to enhance feature robustness; Hyperparameters are used to balance the contributions of each component.

[0231] In this embodiment, The aim is to provide clear gradient guidance for shallow networks in the early stages of training, thereby accelerating model convergence and improving the accuracy of topological features.

[0232] The different loss functions are constructed as follows:

[0233] 1. Main Classification Loss

[0234] The main classification loss includes classification loss. and bounding box regression loss Its construction method is as follows:

[0235] (1) Classification loss

[0236] The majority of areas in a circuit board image are background, with only a very small number containing defects (such as open or short areas). Standard cross-entropy loss is easily dominated by gradients from a large number of simple background regions, making it difficult for the model to learn the features of these minute defects. Therefore, this invention uses the existing Focal Loss as the classification loss function, expressed as:

[0237]

[0238] Where M represents the total number of samples in the feature map; C represents the number of categories; p m,c γ is the probability that the m-th sample belongs to class c; γ is the focusing parameter, with a value of 2.0, which reduces the weight contribution of easily classified samples when calculating the loss, forcing the model to focus on the difficult-to-classify samples during training; α c As a category balancing parameter, a larger α value is assigned to defect categories that occur less frequently or are more difficult to detect, in order to increase their contribution to the gradient.

[0239] (2) Bounding Box Regression Loss

[0240] To address the characteristics of PCB defects, which typically present as elongated lines (such as mouse bites and open areas) and exhibit significant dimensional variations, this invention employs the existing CIoU (Complete IoU) loss to optimize the bounding box coordinate regression. The CIoU loss, based on IoU, also considers overlap area, center point distance, and aspect ratio consistency; its expression is as follows:

[0241]

[0242] Where IoU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box; ρ(*) represents the Euclidean distance; c pred and c gt λ represents the center points of the predicted and ground truth bounding boxes, respectively; d is the diagonal distance of the minimum closure region covering both boxes; α and v are penalty terms used to measure aspect ratio consistency. This loss function is used in the regression branches of the main detection head and the graph network auxiliary head.

[0243] (2) Graph auxiliary loss

[0244] In the intermediate layer feature extraction structure of the network, the Dynamic Difference Graph Convolutional Network (DDGAN) aims to capture long-range dependencies between PCB components through graph topology. To ensure that the supernodes in the graph can accurately aggregate features of defect regions and prevent the gradient vanishing problem during deep network training, this invention designs intermediate supervision in this layer. That is, a lightweight auxiliary detection branch is added to the output features of the second-layer DDGAN to directly predict the category and location of graph node / supernode features and calculate the graph auxiliary loss L. graph This auxiliary loss propagates back through the main detection loss during training, and its gradient can directly apply to DNFM and DDGAN. This results in defect discrimination in the mid-layer (constraint node representation), stable semantic aggregation of supernodes, and improved input quality for subsequent deep features in SSEFormer. The output features of the graph network... Inputting the data into a lightweight auxiliary detection head yields the predicted class probability P of the j-th node. j graph With predicted bounding box b j graph, And construct auxiliary loss of the coating surface. Its expression is:

[0245]

[0246] Where, N node The number of nodes required to construct a graph network; and These are the predicted class probability and predicted bounding box of the j-th node, respectively; and The true label is determined based on the matching of node centers with the ground truth bounding boxes. This auxiliary loss forces the graph network to learn the topological distribution of defects in the intermediate layers, thereby providing more discriminative input features for the deep feature extraction module.

[0247] Step 4: Acquire images of the circuit board under test using the acquisition module. Scale and pad the original frames to the model input size (640×640) using letterboxing / proportional scaling + padding. Perform color space conversion, normalization, and tensor arrangement (HWC→CHW) to ensure consistency with the input distribution during training. Input the pre-processed image into the trained defect detection model for inference to obtain the defect category confidence and bounding box prediction for the circuit board. Overlay the detection boxes and category labels onto the original image frames in real time and display the circuit board defect detection results on the screen in real time.

Claims

1. A method for detecting defects in circuit boards incorporating cascaded heterogeneous networks, characterized in that: The method includes: A defect detection model is constructed; the defect detection model includes a shallow feature extraction module, a middle feature extraction module, and a deep feature extraction module connected in sequence; the shallow feature extraction module, the middle feature extraction module, and the deep feature extraction module process defect features from local to global and from structure to semantics in sequence; The mid-layer feature extraction module includes a reshaping module, a dynamic node feature generation module, a dynamic difference graph attention module, and a post-processing module connected in sequence. The reshaping module is used to adjust the size of the shallow feature map. The dynamic node feature generation module is used to cluster the pixel-level features in the output feature map of the reshaping module into multiple nodes and construct node features and edges between nodes. The dynamic difference graph attention module is used to obtain the relative change information between nodes based on the edges between nodes and integrate the relative change information into the node's own features. The post-processing module is used to constrain the output feature map of the dynamic difference graph attention module to obtain the mid-layer feature map. Acquire images of the circuit board under test and preprocess the images; input the images of the circuit board under test into the defect detection model for defect detection.

2. The circuit board defect detection method incorporating a cascade heterogeneous network according to claim 1, characterized in that: In the shallow feature extraction module, the input feature map is processed by multiple cascaded depthwise separable convolutional blocks, and the processing results are fused with the input feature map processed by the convolutional layer to obtain a spatial local feature map. The spatial local feature map is processed by a dual-path cross-cooperative attention module, and the processing results are fused with the input feature map processed by the convolutional layer to obtain the shallow feature map output by the shallow feature extraction module.

3. The circuit board defect detection method incorporating a cascade heterogeneous network according to claim 2, characterized in that: In the dual-path cross-cooperative attention module, convolutional layers of different sizes are used to process the input feature map to obtain two feature response values. After performing global average pooling on the two feature response values, the features are concatenated to obtain a fused feature map. A multilayer perceptron is used to convert the fused feature map into importance weights, and the importance weights are split into two dynamic weights. Two multilayer perceptrons are used to process the two dynamic weights respectively. After reshaping the processing results, features are extracted from the input feature map to obtain two feature representations. The two feature representations are fused to obtain a channel-level feature map. Average pooling and max pooling operations are applied to the channel-level feature map respectively and then fused to obtain the feature descriptor. The spatial attention map is extracted from the feature descriptor using depthwise separable convolution, and the two feature response values ​​are weighted and fused based on the spatial attention map to obtain the shallow feature map extracted by the shallow feature extraction module.

4. The circuit board defect detection method incorporating a cascade heterogeneous network according to claim 1, characterized in that: In the dynamic node feature generation module, a neural network is used to obtain the soft attribution matrix from the shallow feature map to multiple supernodes; based on the soft attribution matrix, the feature pixels in the shallow feature map are aggregated, and the node features and center coordinates of each node are obtained according to the aggregation result; the center coordinates of each node are used to construct the edges between nodes.

5. The circuit board defect detection method incorporating a cascade heterogeneous network according to claim 1, characterized in that: The dynamic difference graph attention module includes multiple stacked difference graph attention layers; Each difference graph attention layer updates the input node features based on the product of the difference features and the attention coefficients; The differential feature is the difference between the features of different nodes; The attention coefficients are constructed based on the differential features and similarity between different nodes.

6. The circuit board defect detection method incorporating a cascade heterogeneous network according to claim 1, characterized in that: In the deep feature extraction module, two cascaded state-space enhanced self-attention encoders are used to process the input feature map to obtain a first intermediate feature map. The result of the entropy-modulated quantum residual module processing of the intermediate feature map is fused with the first intermediate feature map, and the fused result is processed by the state-space enhanced self-attention encoder to obtain a second intermediate feature map. The result of the entropy-modulated quantum residual module processing of the output feature map of the first state-space enhanced self-attention encoder is fused with the second intermediate feature map to obtain a fused feature map. The detection head is used to process the fused feature map to obtain the output of the defect detection model.

7. The circuit board defect detection method incorporating a cascade heterogeneous network according to claim 6, characterized in that: In the state-space augmented self-attention encoder, the input feature map is fused with the location embedding to obtain the feature map. Enhanced self-attention module for feature maps using state space The processing is performed, and the processing results are compared with the feature map. After fusion, a normalization operation is performed to obtain a fused feature map. The fused feature map is then processed by a feedforward neural network, and the processing result is fused with the fused feature map before a layer normalization operation is performed to obtain the output feature map of the state-space enhanced self-attention encoder.

8. The circuit board defect detection method incorporating a cascade heterogeneous network according to claim 6, characterized in that: In the state-space enhanced self-attention module, the feature map Project them into a query matrix, a key matrix, and a value matrix, respectively; The value matrix is ​​processed by a state-space model, and the processing result is then weighted and fused with the value matrix to obtain the final value matrix. The output feature map of the state-space enhanced self-attention module is obtained by calculating the attention weights of the query matrix, key matrix, and final value matrix.

9. The circuit board defect detection method incorporating a cascade heterogeneous network according to claim 6, characterized in that: The entropy-modulated quantum residual block constructs the information barrier penetration probability based on the difference between the input feature distribution and the target distribution, and uses the product of the information barrier penetration probability and the input feature map as the output feature map.

10. A circuit board defect detection system integrating a cascaded heterogeneous network, characterized in that: This system is used to perform a circuit board defect detection method incorporating a cascaded heterogeneous network as described in claim 1. The circuit board defect detection system includes an image acquisition module, a preprocessing module, a defect detection module, and a visualization module. The image acquisition module is used to acquire images of the circuit board under test. The preprocessing module is used to scale, fill, convert color space, normalize, and perform tensor arrangement on the image under test. The defect detection module is used to detect defects in the preprocessed image. The visualization module is used to overlay the detection box and category label output by the defect detection module onto the image under test and display it on a display screen.

Images