Automatic Identification System and Method for Anomaly Points in Gold Wire Bonding Based on Neural Network Architecture

Abstract

The present invention discloses a system and method for automatically identifying abnormal points in gold wire bonding based on a neural network architecture, and relates to the technical field of image processing. The inspection image of the product is obtained by taking a picture with an electron microscope, and is input into the backbone network. After multiple convolution and pooling operations, features of different scales are extracted; then, in the hybrid encoder, after a series of convolution and upsampling operations, the feature map size is adjusted to perform feature fusion; then, through a downsampling operation, it is spliced with the previously fused feature map, and feature fusion is performed again. Finally, the feature maps of three different scales are input into the decoder for target detection, thereby completing the automatic identification of gold wires. The product image is processed by the improved RT-DETR model to achieve automatic gold wire identification and defect labeling, so that the operator can quickly locate the defect based on the defect labeling and complete the gold wire re-insertion operation, thereby improving production efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a system and method for automatically identifying abnormal points in gold wire bonds based on a neural network architecture. Background Art

[0002] Wire bonding is a critical step in microassembly production and is crucial to achieving product functionality and performance. Complex RF MCMs (multi-chip modules), such as the TR (transmitter / receiver) components of phased array systems, typically contain numerous bare chips. To ensure signal transmission, these bare chips, as well as the bare chips and microwave PCB substrates, require wire bonding via gold wire interconnects. Due to the large number of chips and complex interconnects within RF MCM modules, wire bonding quality is crucial for ensuring stable signal transmission and overall module performance. In mass production, to improve efficiency, companies typically use fully automatic wire bonding machines. However, in actual production, wire skipping is a common problem, resulting in some bond pads failing to complete the wire connection. This problem can directly impact the product's electrical performance and consistency, and may even cause functional failures in subsequent processes.

[0003] Faced with the problem of wire jumps, small and medium-sized enterprises often rely on manual identification of wire jump locations and then use manual bonding machines to re-bond the wires. However, this manual identification method has significant limitations. In complex RF MCM modules, due to the large number and dense distribution of bond points, operators often struggle to accurately identify all wire jump locations, making it easy to miss or misjudge them. Furthermore, staring at wire bond points for extended periods can cause eye fatigue and decreased concentration, which in turn affects identification accuracy and re-bonding efficiency. Due to the inefficiency of wire jump detection and repair, the overall process time increases, production efficiency decreases, and ultimately affects the overall performance of the microassembly production line. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of low production efficiency mentioned in the above background technology, and to propose a system and method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture.

[0005] A first aspect of the present invention provides a method for automatically identifying abnormal points in gold wire bonds based on a neural network architecture, the method comprising:

[0006] Step S101, obtaining a detection image of a target product;

[0007] Step S102, performing feature extraction on the detection image to obtain a multi-scale feature map;

[0008] Step S103, performing cross-scale channel fusion on the multi-scale feature maps to obtain a fused feature map;

[0009] Step S104: performing target detection based on the fused feature map to obtain a gold wire detection result; the detection result includes gold wire position information;

[0010] Step S105, matching the detection result with a preset standard position, and marking the abnormal position of the gold wire bonding on the detection image;

[0011] Wherein, step S102, step S103 and step S104 are implemented by a pre-trained gold wire recognition model; the gold wire recognition model is a model improved based on the RT-DETR neural network architecture.

[0012] Preferably, the RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder;

[0013] Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements:

[0014] The preset Bottleneck_DySnake module is embedded at the end of the backbone network to obtain an improved feature extraction network; the calculation process of the Bottleneck_DySnake module includes:

[0015] First, the input data is extracted through a 1×1 convolution layer, and then passed to the dynamic snake convolution layer for targeted feature extraction of slender objects. Then, a 1×1 convolution layer is used to reduce the number of channels and integrate information to obtain the target features.

[0016] If the residual connection is enabled, the original input is added to the target feature and then output; otherwise, the target feature is directly output.

[0017] Preferably, the gold wire recognition model, compared with the original RT-DETR model, further includes the following improvements:

[0018] Applying a preset gated attention upsampling module GMAU to the upsampling process of the hybrid encoder; the GMAU includes a first upsampling branch, a second upsampling branch, and a first attention calculation branch;

[0019] The GMAU calculation process includes:

[0020] The first attention calculation branch first performs global average pooling on the input feature map X1 to obtain the first global context information; then the first global context information is converted into the channel attention weight T1 through a convolutional layer and a Hardsigmoid activation function;

[0021] The first upsampling branch uses transposed convolution to improve the resolution of the feature map X1 to obtain the feature map U1;

[0022] The second upsampling branch uses interpolation to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2;

[0023] After concatenating the feature map U1 and the feature map U2, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 and output the upsampled feature map.

[0024] Preferably, the gold wire recognition model, compared with the original RT-DETR model, further includes the following improvements:

[0025] Applying a preset gated attention downsampling module GMAD to the downsampling process of the hybrid encoder; the GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch;

[0026] The calculation process of the GMAD includes:

[0027] The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into the channel attention weight T2 through a convolutional layer and a Hardsigmoid activation function;

[0028] The first downsampling branch uses a 3×3 convolution with a step size of 2 to perform feature dimensionality reduction and spatial compression on the feature map X2 to obtain a feature map D1;

[0029] The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2;

[0030] After splicing the feature map D1 and the feature map D2, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 and output the downsampled feature map.

[0031] Preferably, the gold wire recognition model, compared with the original RT-DETR model, further includes the following improvements:

[0032] A preset cross-stage parallel dilated rotation equivariant convolution module CSP_RPDC is applied to the cross-scale fusion process of the hybrid encoder. Specifically, the RepC3 module in the hybrid encoder is replaced with the CSP_RPDC; the CSP_RPDC includes a parallel dilated convolution layer and a rotation equivariant convolution layer;

[0033] The calculation process of CSP_RPDC includes:

[0034] The parallel expansion convolution layer uses convolution kernels with different expansion rates to process the input feature map X3 in parallel, captures features of different receptive fields, and performs cross-scale feature interaction through channel splicing and 1×1 convolution layer to obtain the feature map F1;

[0035] The rotational equivariant convolution layer performs convolution operations on the input feature map X3 in multiple directions to enhance the feature expression capability and obtain the feature map F2;

[0036] After concatenating the feature map F1 and the feature map F2, feature fusion is performed through a 1×1 convolutional layer to output the fused feature map.

[0037] A second aspect of the present invention provides a system for automatically identifying abnormal points in gold wire bonds based on a neural network architecture, the system comprising:

[0038] An image acquisition module is used to acquire a detection image of a target product;

[0039] The target detection module is configured to call a pre-trained gold thread recognition model, perform feature extraction on the detection image, and obtain a multi-scale feature map; perform cross-scale channel fusion on the multi-scale feature map to obtain a fused feature map; perform target detection based on the fused feature map to obtain a gold thread detection result; the gold thread recognition model is an improved model based on the RT-DETR neural network architecture; the detection result includes gold thread location information;

[0040] The defect marking module is used to match the detection result with a preset standard position and mark the abnormal position of the gold wire bonding on the detection image.

[0041] Preferably, the RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder;

[0042] Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements:

[0043] The preset dynamic snake bottleneck module Bottleneck_DySnake is embedded at the end of the backbone network to obtain an improved feature extraction network; the calculation process of Bottleneck_DySnake includes:

[0044] First, the input data is extracted through a 1×1 convolution layer, and then passed to the dynamic snake convolution layer for targeted feature extraction of slender objects. Then, a 1×1 convolution layer is used to reduce the number of channels and integrate information to obtain the target features.

[0045] If the residual connection is enabled, the original input is added to the target feature and then output; otherwise, the target feature is directly output.

[0046] Preferably, the gold wire recognition model, compared with the original RT-DETR model, further includes the following improvements:

[0047] A gated attention upsampling module (GMAU) is constructed based on the gating mechanism and the attention mechanism, and applied to the upsampling process of the hybrid encoder; the GMAU includes a first upsampling branch, a second upsampling branch, and a first attention calculation branch;

[0048] The calculation process of the gated attention upsampling module includes:

[0049] The first attention calculation branch first performs global average pooling on the input feature map X1 to obtain the first global context information; then the first global context information is converted into the channel attention weight T1 through a convolutional layer and a Hardsigmoid activation function;

[0050] The first upsampling branch uses transposed convolution to improve the resolution of the feature map X1 to obtain the feature map U1;

[0051] The second upsampling branch uses interpolation to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2;

[0052] After concatenating the feature map U1 and the feature map U2, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 and output the upsampled feature map.

[0053] Preferably, the gold wire recognition model, compared with the original RT-DETR model, further includes the following improvements:

[0054] A gated attention downsampling module GMAD is constructed based on the gating mechanism and the attention mechanism, and is applied to the downsampling process of the hybrid encoder; the GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch;

[0055] The calculation process of the GMAD includes:

[0056] The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into the channel attention weight T2 through a convolutional layer and a Hardsigmoid activation function;

[0057] The first downsampling branch uses a 3×3 convolution with a step size of 2 to perform feature dimensionality reduction and spatial compression on the feature map X2 to obtain a feature map D1;

[0058] The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2;

[0059] After splicing the feature map D1 and the feature map D2, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 and output the downsampled feature map.

[0060] Preferably, the gold wire recognition model, compared with the original RT-DETR model, further includes the following improvements:

[0061] According to the dilated convolution and rotational equivariant convolution, a cross-stage parallel dilated rotational equivariant convolution module CSP_RPDC is constructed, and the RepC3 module in the hybrid encoder is replaced with the CSP_RPDC; the CSP_RPDC includes a parallel dilated convolution layer and a rotational equivariant convolution layer;

[0062] The calculation process of CSP_RPDC includes:

[0063] The parallel expansion convolution layer uses convolution kernels with different expansion rates to process the input feature map X3 in parallel, captures features of different receptive fields, and performs cross-scale feature interaction through channel splicing and 1×1 convolution layer to obtain the feature map F1;

[0064] The rotational equivariant convolution layer performs convolution operations on the input feature map X3 in multiple directions to enhance the feature expression capability and obtain the feature map F2;

[0065] After concatenating the feature map F1 and the feature map F2, feature fusion is performed through a 1×1 convolutional layer to output the fused feature map.

[0066] Beneficial effects of the present invention:

[0067] The present invention proposes a method for automatically identifying abnormal gold wire bonding points based on a neural network architecture. The method includes: acquiring a test image of a target product; calling a pre-trained gold wire recognition model to extract features from the test image to obtain a multi-scale feature map; performing cross-scale channel fusion on the multi-scale feature map to obtain a fused feature map; performing target detection based on the fused feature map to obtain a gold wire detection result; the gold wire recognition model is an improved model based on the RT-DETR neural network architecture; the detection result includes gold wire position information; matching is performed based on the detection result and a preset standard position, and the abnormal gold wire bonding position is marked on the test image.

[0068] By processing product images through the improved RT-DETR model, automated gold wire recognition and defect marking are achieved, allowing operators to quickly locate the defects based on the defect markings and guide the completion of the gold wire re-insertion operation, reducing the time for defect identification and repair and improving production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of a method for automatically identifying abnormal points in gold wire bonding based on a neural network architecture is provided for an embodiment of the present invention;

[0070] Figure 2 A network architecture diagram of a gold wire recognition model is provided for an embodiment of the present invention;

[0071] Figure 3 A structural schematic diagram of a dynamic serpentine bottleneck structure is provided for an embodiment of the present invention;

[0072] Figure 4 A schematic diagram of the structure of a gated attention upsampling module is provided for an embodiment of the present invention;

[0073] Figure 5 A schematic structural diagram of a cross-stage parallel expansion and rotation equivariant convolution module is provided for an embodiment of the present invention;

[0074] Figure 6 A schematic diagram of the structure of a gated attention downsampling module is provided for an embodiment of the present invention;

[0075] Figure 7 A schematic diagram comparing training results before and after model improvement is provided for an embodiment of the present invention;

[0076] in: Figure 7 (a) shows a schematic diagram of the mAP50-95 average precision training of the original RT-DETR model;

[0077] Figure 7 (b) is a diagram showing the mAP50-95 mean average precision training of the gold wire recognition model proposed in the present invention;

[0078] Figure 8 Another schematic diagram comparing training results before and after model improvement is provided for an embodiment of the present invention;

[0079] in: Figure 8 (a) is a schematic diagram showing the gold wire confidence of the original RT-DETR model in densely packed and difficult-to-identify areas;

[0080] Figure 8(b) is a schematic diagram showing the gold wire confidence of the gold wire recognition model proposed in the present invention in a densely arranged and difficult to recognize area;

[0081] Figure 9 An embodiment of the present invention provides an architecture diagram of a system for automatically identifying abnormal points in gold wire bonds based on a neural network architecture;

[0082] Figure 10 A defect marking schematic diagram is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0084] The embodiment of the present invention provides a method for automatically identifying abnormal points in gold wire bonding based on a neural network architecture. Figure 1 , Figure 1 A flowchart of a method for automatically identifying abnormal points in gold wire bonds based on a neural network architecture is provided in an embodiment of the present invention. The method includes the following steps:

[0085] S101, obtaining a detection image of a target product.

[0086] S102: Extract features from the detection image to obtain a multi-scale feature map.

[0087] S103: Perform cross-scale channel fusion on the multi-scale feature maps to obtain a fused feature map.

[0088] S104: Perform target detection based on the fused feature map to obtain a gold wire detection result.

[0089] S105 , matching is performed based on the detection result and the preset standard position, and the abnormal position of the gold wire bonding is marked on the detection image.

[0090] The detection results include gold wire position information.

[0091] According to an embodiment of the present invention, a method for automatically identifying gold wire bonding anomalies based on a neural network architecture is provided. Product images are processed through an improved RT-DETR model to achieve automated gold wire recognition and defect labeling, thereby enabling operators to quickly locate defects based on defect labeling and provide guidance for completing the gold wire re-insertion operation, reducing the time for defect identification and repair and improving production efficiency.

[0092] In one embodiment, steps S102, S103, and S104 are implemented using a pre-trained gold thread recognition model; the gold thread recognition model is an improved model based on the RT-DETR neural network architecture. The RT-DETR neural network architecture includes a backbone network, a hybrid encoder, and a decoder. The backbone network is used to extract feature representations of the input image; the hybrid encoder is used to convert multi-scale features into image feature sequences through intra-scale feature interaction and cross-scale feature fusion; and the decoder is used to perform object detection and generate bounding boxes and confidence scores.

[0093] See also Figure 2 , Figure 2 The network architecture diagram of a gold wire recognition model is provided for an embodiment of the present invention. As shown in the figure, the backbone network and hybrid encoder are modified. Specifically:

[0094] 1. Propose a self-developed Bottleneck_DySnake module.

[0095] Based on the residual bottleneck structure, the second convolution layer is replaced by dynamic snake convolution to obtain the dynamic snake bottleneck module Bottleneck_DySnake. Figure 3 , Figure 3 A schematic diagram of a dynamic snake bottleneck structure provided by an embodiment of the present invention. The computational process of the Bottleneck_DySnake module includes: first, feature extraction of the input data is performed through a 1×1 convolutional layer, which is then passed to a dynamic snake convolutional layer for targeted feature extraction of slender objects. Next, a 1×1 convolutional layer is used to reduce the number of channels and integrate the information to obtain the target features. If residual connections are enabled (shortcut=true), the original input is summed with the target features before output; otherwise, the target features are directly output.

[0096] The dynamic snake convolution layer contains a three-branch structure consisting of a standard convolution Conv3×3 (followed by batch normalization BN and ReLU activation function), an x-axis dynamic snake convolution DSConv_X (followed by group normalization GN and ReLU activation function), and a y-axis dynamic snake convolution DSConv_Y (followed by group normalization GN and ReLU activation function). The outputs of the three branches are spliced and input into the next layer.

[0097] In one implementation, the Bottleneck_DySnake module is embedded at the end of the backbone network to obtain an improved feature extraction network. Specifically, the improved feature extraction network consists of three ConvNormLayer modules, one MaxPool layer, three Bottleneck blocks modules, and one Bottleneck_DySnake blocks module. Specifically:

[0098] First, the network architecture employs three ConvNormLayer modules. These modules are responsible for performing preliminary feature extraction on the input image. Each ConvNormLayer module consists of a convolutional layer, a normalization layer, and a ReLU activation function. The convolutional layer extracts local features using a 3×3 convolution kernel. The normalization layer helps accelerate model training and improve generalization, while the activation function enhances the model's nonlinear expression capabilities. These three modules gradually increase the number of channels, from 32 to 64, enabling the network to capture richer image features.

[0099] Next, the network introduces a MaxPool layer, which reduces the size of the feature map by half through a 3×3 pooling kernel with a step size of 2, providing a more compact feature representation for the subsequent deep network.

[0100] The network then includes three Bottleneck blocks, a highly efficient feature extraction architecture. Each Bottleneck block consists of multiple Bottleneck units. Each unit uses 1×1, 3×3, and 1×1 convolutional layers to achieve cross-channel feature fusion and dimensionality reduction, effectively reducing the number of parameters and computation. In one implementation, a feasible configuration is: the first, second, and third Bottleneck blocks contain 3, 4, and 6 Bottleneck units, respectively.

[0101] Finally, to address the long, thin nature of the gold wire, the network introduced a self-developed module, Bottleneck_DySnakeblocks. In one implementation, a feasible configuration is: the Bottleneck_DySnake blocks module contains three Bottleneck_DySnake modules.

[0102] The advantages of the entire backbone network structure are mainly reflected in the following aspects: 1. Hierarchical feature extraction: From shallow to deep layers, the network gradually extracts abstract features of the image, which is conducive to subsequent task recognition. 2. Parameter efficiency: By using the Bottleneck structure and 1×1 convolution, the network reduces the number of parameters while ensuring the richness of features. 3. Multi-scale feature fusion: The MaxPool layer and multiple Bottleneck blocks modules realize multi-scale feature fusion, enabling the network to cope with targets of different scales. 4. Dynamic convolution: The introduction of dynamic snake convolution makes the network more adaptable to gold wire bonding scenarios.

[0103] 2. Propose a self-developed gated attention upsampling module (GMAU, Gate Mechanism Attention Upsample). Figure 4 , Figure 4 A schematic diagram of the structure of a gated attention upsampling module provided in an embodiment of the present invention. The GMAU includes a first upsampling branch, a second upsampling branch, and a first attention calculation branch.

[0104] The GMAU calculation process includes:

[0105] The first attention calculation branch first performs global average pooling (GAP) on the input feature map X1 to obtain the first global context information; then the first global context information is converted into the channel attention weight T1 through a convolutional layer and a Hardsigmoid activation function;

[0106] The first upsampling branch uses transposed convolution (ConvTranspose) to improve the resolution of feature map X1 and obtain feature map U1;

[0107] The second upsampling branch uses interpolation to upsample (Upsample) to increase the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2;

[0108] After concatenating the feature map U1 and the feature map U2, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3. Finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 and output the upsampled feature map.

[0109] In one implementation, the gated attention upsampling module is applied to the upsampling process of the hybrid encoder. The parallel upsampling branches provide the network with multiple feature extraction pathways, enriching the diversity of feature representation. Combined with the gating mechanism, the sampled features are selected to enhance more meaningful and relevant features while suppressing redundant or irrelevant ones, thereby improving the effectiveness of feature representation.

[0110] 3. We propose a self-developed CrossStage Partial_ Rotation-equivariant Parallel Dilated Convolution (CSP_RPDC). Figure 5 , Figure 5This is a schematic diagram of the structure of a cross-stage parallel dilated and rotated equivariant convolution module provided by an embodiment of the present invention. CSP_RPDC includes a parallel dilated convolution layer (PDC, ParallelDilatedConv) and a rotation-equivariant convolution layer (RotC);

[0111] The calculation process of CSP_RPDC includes:

[0112] The parallel dilated convolution layer (PDC) uses dilated convolutions (DilatedConv) with different dilation rates to process the input feature map X3 in parallel, capturing features from different receptive fields. After concatenating features at different scales according to the channel dimension, a 1×1 convolutional layer performs cross-scale feature interaction to generate the feature map F1. Specifically, three sets of convolution kernels with dilation rates of 1, 2, and 3 are used to simultaneously capture the local texture, mid-range topology, and global morphological features of the gold wire target.

[0113] The rotational equivariant convolution layer, RotC, performs convolution operations on the input feature map X3 in multiple directions to enhance the expressiveness of the features, resulting in the resulting feature map F2. Since gold wire detection can occur at various rotation angles, RotC, which is rotationally invariant, is specifically selected for identification. Specifically, the rotational convolution performs rotations of the input features in four directions: 0°, 90°, 180°, and 270°. The convolution weights are shared to extract direction-independent features, effectively addressing the arbitrary angle distribution of gold wire targets in the image.

[0114] After concatenating the feature map F1 and the feature map F2, feature fusion is performed through a 1×1 convolutional layer to output the fused feature map.

[0115] In one implementation, the cross-stage parallel dilated rotation equivariant convolution module (CSP_RPDC) is applied to the cross-scale fusion process of a hybrid encoder. Specifically, the RepC3 module in the hybrid encoder is replaced with CSP_RPDC. The advantages of this entire module are: first, it can extract multi-directional features. The RotC layer can identify directional features in the image, improving the detection of gold filaments that are sensitive to rotation or direction. Second, it can perform multi-scale feature fusion. The PDC and RotC layers work together to enable the module to adapt to gold filaments of different scales, improving detection accuracy.

[0116] 4. Propose a self-developed gated attention downsampling module (GMAD, Gate Mechanism Attention Downsample). Figure 6 , Figure 6A schematic diagram of the structure of a gated attention downsampling module provided in an embodiment of the present invention. GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch;

[0117] The GMAD calculation process includes:

[0118] The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into the channel attention weight T2 through a convolutional layer and a Hardsigmoid activation function;

[0119] The first downsampling branch uses a 3×3 convolution with a stride of 2 to perform feature dimensionality reduction and spatial compression on the feature map X2 to obtain the feature map D1;

[0120] The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2;

[0121] After concatenating the feature map D1 and the feature map D2, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 and output the downsampled feature map.

[0122] In one implementation, the gated attention downsampling module (GMAD) is applied to the downsampling process of the hybrid encoder. It leverages global context to weight features during the downsampling process, preserving more important information. This design helps improve the model's robustness and generalization capabilities in complex scenarios. By combining the gating mechanism with the downsampling operation, the module effectively extracts and fuses features, supporting subsequent object detection tasks.

[0123] In one embodiment, the proposed modules are embedded in the RT-DETR model to produce a gold thread recognition model. The backbone network, an improved feature extraction network, adopts a progressive feature extraction architecture to achieve multi-scale feature learning. The hybrid encoder performs feature enhancement through a bidirectional interaction between top-down and bottom-up approaches, and the last three layers of CSP_RPDC modules output fused feature maps. The fused multi-scale feature maps are fed into the RT-DETR decoder, which is responsible for generating the final detection results. In the decoder, the feature maps are further processed to generate predictions for the gold thread's location and bounding box.

[0124] In one embodiment, the entire training process uses a backpropagation algorithm to guide the update of network parameters by calculating loss functions such as classification loss, localization loss, and bounding box regression loss. This process is iterated until the network reaches the predetermined performance indicator, thus completing model training.

[0125] The model training process is as follows:

[0126] First, the dataset must be collected and created. This process builds a dataset based on 880 collected gold wire image samples. The image samples (including the gold wire targets) in the dataset are annotated, generating annotation files corresponding to the image samples. Furthermore, the training dataset is enhanced through methods such as color dithering, flipping, scaling and cropping, Gaussian blurring, and Gaussian noise injection.

[0127] The training set, test set, and validation set are then divided into a 7:1:2 ratio. The training set is used to train the model, which adjusts its parameters using the training set data to minimize the loss function. The validation set is used to evaluate the model's performance during training and guide the adjustment and selection of model hyperparameters. The test set is used for the final evaluation of model performance after the model is fully trained to help verify its performance on unseen data.

[0128] During training, a dynamic label assignment strategy is employed, using the Hungarian algorithm to optimize the match between predicted and ground-truth bounding boxes. A composite loss function is also designed: Focal Loss is used for classification to address class imbalance, while L1 and GIoU losses are used for regression to jointly monitor bounding box accuracy. A directional consistency constraint loss is introduced to account for the slender nature of gold wire. When updating parameters via gradient backpropagation, the model employs an adaptive learning rate scheduling strategy. Initially, training focuses on the low-level feature extraction network, while in later stages, the coordinated optimization of high-level semantic features and the decoder's attention mechanism is gradually strengthened. Ultimately, the model possesses the ability to stably detect micron-level gold wires in complex industrial scenarios.

[0129] The gold wire bonding anomaly recognition system proposed in the present invention is highly robust. Even if the image is affected by factors such as rotation, scaling or lighting changes, the system can still relatively stably match key points and identify differences. It has strong robustness and can adapt to different image scenes.

[0130] Training results and improvement before and after comparison see Figure 7 and Figure 8 . Figure 7 (a) shows the mAP50-95 mean average precision of the original RT-DETR model (the corresponding mAP50-95 mean average precision means: after 200 rounds of training, the intersection-over-union ratio threshold is 0.5 to 0.95, and the interval is 0.05 to obtain 10 mAP values, and then take the average of the average precision); Figure 7 (b) shows the mAP50-95 mean average precision of the gold wire recognition model proposed in this paper. The mAP50-95 mean average precision of gold wire stable recognition increased from 0.574 to 0.671. Figure 8 (a) shows the gold wire confidence of the original RT-DETR model in densely packed and difficult-to-identify areas; Figure 8 (b) shows the gold wire confidence of the gold wire recognition model proposed in the present invention in the densely arranged and difficult-to-recognize area; after the improvement, the confidence of gold wire recognition is increased from 71% to 84%.

[0131] The embodiment of the present invention provides a system for automatically identifying abnormal points in gold wire bonding based on a neural network architecture. Figure 9 , Figure 9 This is a diagram of the architecture of a system for automatically identifying abnormal gold wire bonding points based on a neural network architecture, provided in an embodiment of the present invention. The system includes:

[0132] The image acquisition module is used to acquire the detection image of the target product.

[0133] The target detection module is used to call the pre-trained gold wire recognition model to extract features from the detection image to obtain a multi-scale feature map; the multi-scale feature map is fused across scale channels to obtain a fused feature map; and target detection is performed based on the fused feature map to obtain the gold wire detection result.

[0134] The defect marking module matches the inspection results with the preset standard positions and marks the abnormal positions of the gold wire bonding on the inspection image.

[0135] Among them, the gold wire recognition model is a model improved based on the RT-DETR neural network architecture; the detection results include gold wire location information.

[0136] According to an embodiment of the present invention, a system for automatically identifying gold wire bonding anomalies based on a neural network architecture is provided. Product images are processed through an improved RT-DETR model to achieve automatic gold wire recognition and defect labeling, so that operators can quickly locate the defects based on the defect labels and are guided to complete the gold wire re-insertion operation, thereby reducing the time for defect identification and repair and improving production efficiency.

[0137] In one embodiment, the system hardware for applying the gold wire recognition model includes an electron microscope (with a photographing function) and a computer (meeting the running requirements of the gold wire defect recognition program), and the software includes an automatic gold wire defect recognition program.

[0138] The process of automatic identification of gold wire bonding abnormal points based on the neural network architecture is as follows:

[0139] During the batch production of RF MCM modules, the first piece completes all steps, including wire bonding, according to the process flow. This means the microwave PCB substrate is sintered, surface-mount components are soldered, and the bare chip carrier is sintered. All bare chips and the microwave PCB are installed. After manual confirmation that the first piece is assembled correctly, the first piece is sent to an inspection station for an electron microscope image. This image is then entered into a computer to establish a comparison baseline, serving as the standard image of a wire-bonded module free of wire jump defects. This then enters the batch production phase of the RF MCM modules. After fully automated wire bonding is complete, the batch production modules are sent to an inspection station for an electron microscope image, which is then entered into a computer.

[0140] 2. Import standard images of gold wire bonding modules without wire skipping defects, use the mature trained model to accurately identify these standard images, and obtain the standard position, that is, the detection frame position of the standard image.

[0141] 3. Detect and analyze the images of batch products. This process uses the gold wire recognition model to output the coordinates of all gold wires in the photos of batch products. Map the standard position to the image of the batch product. If the gold wire cannot be detected near the corresponding coordinates in the batch product image, for example, the coordinate deviation of the four corners of the detection frame reaches 20 pixels, it is determined that there is a wire jump error at that position in the batch product image. Mark the position where the wire jump error occurs with a blue frame and label it "LOSS", such as Figure 10 As shown, this ensures accurate monitoring of wire bonding quality and effective identification of defects, thus guiding personnel through re-bonding operations. Operators can quickly identify and locate the "LOSS" label and are guided through the re-bonding process. After re-bonding is complete, a second photo and identification should be taken to confirm that all missed wire locations have been re-bonded.

[0142] It should be noted that, in this document, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements that are inherent to such process, method, article or apparatus.

[0143] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for automatically identifying abnormal points in gold wire bonding based on a neural network architecture, characterized in that: The method comprises: Step S101, obtaining a detection image of a target product; Step S102, performing feature extraction on the detection image to obtain a multi-scale feature map; Step S103, performing cross-scale channel fusion on the multi-scale feature maps to obtain a fused feature map; Step S104: performing target detection based on the fused feature map to obtain a gold wire detection result; the detection result includes gold wire position information; Step S105, matching the detection result with a preset standard position, and marking the abnormal position of the gold wire bonding on the detection image; Steps S102, S103, and S104 are implemented by a pre-trained gold wire recognition model; the gold wire recognition model is an improved model based on the RT-DETR neural network architecture, and the specific improvements include: A preset cross-stage parallel dilated rotation equivariant convolution module CSP_RPDC is applied to the cross-scale fusion process of the hybrid encoder. Specifically, the RepC3 module in the hybrid encoder is replaced with the CSP_RPDC; the CSP_RPDC includes a parallel dilated convolution layer and a rotation equivariant convolution layer; The calculation process of CSP_RPDC includes: The parallel expansion convolution layer uses convolution kernels with different expansion rates to process the input feature map X3 in parallel, captures features of different receptive fields, and performs cross-scale feature interaction through channel splicing and 1×1 convolution layer to obtain the feature map F1; The rotational equivariant convolution layer performs convolution operations on the input feature map X3 in multiple directions to enhance the feature expression capability and obtain the feature map F2; After concatenating the feature map F1 and the feature map F2, feature fusion is performed through a 1×1 convolutional layer to output the fused feature map.

2. The method for automatically identifying abnormal gold wire bonding points based on a neural network architecture according to claim 1, characterized in that: The RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder; Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: The preset dynamic snake bottleneck module Bottleneck_DySnake is embedded at the end of the backbone network to obtain an improved feature extraction network; the calculation process of Bottleneck_DySnake includes: First, the input data is extracted through a 1×1 convolution layer, and then passed to the dynamic snake convolution layer for targeted feature extraction of slender objects. Then, a 1×1 convolution layer is used to reduce the number of channels and integrate information to obtain the target features. If the residual connection is enabled, the original input is added to the target feature and then output; otherwise, the target feature is directly output.

3. The method for automatically identifying abnormal gold wire bonding points based on a neural network architecture according to claim 2, characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: Applying a preset gated attention upsampling module GMAU to the upsampling process of the hybrid encoder; the GMAU includes a first upsampling branch, a second upsampling branch, and a first attention calculation branch; The GMAU calculation process includes: The first attention calculation branch first performs global average pooling on the input feature map X1 to obtain the first global context information; then the first global context information is converted into the channel attention weight T1 through a convolutional layer and a Hardsigmoid activation function; The first upsampling branch uses transposed convolution to improve the resolution of the feature map X1 to obtain the feature map U1; The second upsampling branch uses interpolation to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2; After concatenating the feature map U1 and the feature map U2, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 and output the upsampled feature map.

4. The method for automatically identifying abnormal gold wire bonding points based on a neural network architecture according to claim 3, characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: Applying a preset gated attention downsampling module GMAD to the downsampling process of the hybrid encoder; the GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch; The calculation process of the GMAD includes: The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into the channel attention weight T2 through a convolutional layer and a Hardsigmoid activation function; The first downsampling branch uses a 3×3 convolution with a step size of 2 to perform feature dimensionality reduction and spatial compression on the feature map X2 to obtain a feature map D1; The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2; After splicing the feature map D1 and the feature map D2, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 and output the downsampled feature map.

5. An automatic identification system for gold wire bonding abnormal points based on a neural network architecture, characterized in that: The system comprises: An image acquisition module is used to acquire a detection image of a target product; The target detection module is configured to call a pre-trained gold thread recognition model to perform feature extraction on the detection image to obtain a multi-scale feature map; perform cross-scale channel fusion on the multi-scale feature map to obtain a fused feature map; perform target detection based on the fused feature map to obtain a gold thread detection result; the detection result includes gold thread position information; A post-processing module performs matching based on the detection result and a preset standard position, and marks the abnormal position of the gold wire bonding on the detection image; The gold wire recognition model is an improved model based on the RT-DETR neural network architecture; the specific improvements include: According to the dilated convolution and rotational equivariant convolution, a cross-stage parallel dilated rotational equivariant convolution module CSP_RPDC is constructed, and the RepC3 module in the hybrid encoder is replaced with the CSP_RPDC; the CSP_RPDC includes a parallel dilated convolution layer and a rotational equivariant convolution layer; The calculation process of CSP_RPDC includes: The parallel expansion convolution layer uses convolution kernels with different expansion rates to process the input feature map X3 in parallel, captures features of different receptive fields, and performs cross-scale feature interaction through channel splicing and 1×1 convolution layer to obtain the feature map F1; The rotational equivariant convolution layer performs convolution operations on the input feature map X3 in multiple directions to enhance the feature expression capability and obtain the feature map F2; After concatenating the feature map F1 and the feature map F2, feature fusion is performed through a 1×1 convolutional layer to output the fused feature map.

6. The system for automatically identifying abnormal gold wire bonding points based on a neural network architecture according to claim 5, characterized in that: The RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder; Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: The preset dynamic snake bottleneck module Bottleneck_DySnake is embedded at the end of the backbone network to obtain an improved feature extraction network; the calculation process of Bottleneck_DySnake includes: First, the input data is extracted through a 1×1 convolution layer, and then passed to the dynamic snake convolution layer for targeted feature extraction of slender objects. Then, a 1×1 convolution layer is used to reduce the number of channels and integrate information to obtain the target features. If the residual connection is enabled, the original input is added to the target feature and then output; otherwise, the target feature is directly output.

7. The system for automatically identifying abnormal gold wire bonding points based on a neural network architecture according to claim 6, characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: A gated attention upsampling module (GMAU) is constructed based on the gating mechanism and the attention mechanism, and applied to the upsampling process of the hybrid encoder; the GMAU includes a first upsampling branch, a second upsampling branch, and a first attention calculation branch; The calculation process of the gated attention upsampling module includes: The first attention calculation branch first performs global average pooling on the input feature map X1 to obtain the first global context information; then the first global context information is converted into the channel attention weight T1 through a convolutional layer and a Hardsigmoid activation function; The first upsampling branch uses transposed convolution to improve the resolution of the feature map X1 to obtain the feature map U1; The second upsampling branch uses interpolation to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2; After concatenating the feature map U1 and the feature map U2, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 and output the upsampled feature map.

8. The system for automatically identifying abnormal gold wire bonding points based on a neural network architecture according to claim 7, characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: A gated attention downsampling module GMAD is constructed based on the gating mechanism and the attention mechanism, and is applied to the downsampling process of the hybrid encoder; the GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch; The calculation process of the GMAD includes: The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into the channel attention weight T2 through a convolutional layer and a Hardsigmoid activation function; The first downsampling branch uses a 3×3 convolution with a step size of 2 to perform feature dimensionality reduction and spatial compression on the feature map X2 to obtain a feature map D1; The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2; After splicing the feature map D1 and the feature map D2, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 and output the downsampled feature map.

Images