Neural network architecture-based gold wire bonding abnormal point automatic identification system and method

Abstract

The invention discloses a gold wire bonding abnormal point automatic identification system and method based on a neural network architecture, and relates to the technical field of image processing. A detection image of a product is obtained through shooting of an electron microscope and input into the backbone network, and features of different scales are extracted through multiple convolution and pooling operations; then, in a hybrid encoder, through a series of convolution and up-sampling operations, adjusting the size of the feature map to carry out feature fusion; and then, through down-sampling operation, splicing with the previously fused feature map, and carrying out feature fusion again. And finally, inputting the three feature maps with different scales into a decoder, and carrying out target detection, thereby completing automatic identification of the gold wire. The product image is processed through the improved RT-DETR model, and automatic gold wire identification and defect marking are realized, so that an operator can quickly position according to the defect marking and complete the gold wire refilling operation, and the production efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a system and method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture. Background Art

[0002] The gold wire bonding process is a key process in the micro-assembly production process and is crucial to the realization of product functional performance. Complex RF MCMs (multi-chip modules) such as phased array system TR (transceiver) components usually contain a large number of bare chips. In order to achieve signal transmission, these bare chips and the bare chips and microwave PCB substrates need to be connected through gold wire interconnections. Due to the large number of chips and complex interconnection points inside the RF MCM module, the quality of gold wire bonding is crucial to ensure the stable transmission of electrical signals and the overall performance of the module. In mass production, in order to improve production efficiency, companies usually use fully automatic gold wire bonding machines to perform the gold wire bonding process. However, in the actual production process, the gold wire bonding process frequently encounters gold wire jumping problems, resulting in the failure of some bonding pads to complete the gold wire connection. This problem will directly affect the electrical performance and consistency of the product, and may even cause functional failures in subsequent processes.

[0003] Faced with the problem of wire jumps, small and medium-sized enterprises often rely on manual identification of wire jump positions and then use manual bonding machines to re-bond the gold wire. However, this manual identification method has significant limitations. In complex RF MCM modules, due to the large number of bonding points and their dense distribution, it is often difficult for operators to accurately identify all wire jump positions, which can easily lead to omissions or misjudgments. In addition, staring at the gold wire bonding points for a long time will cause eye fatigue and decreased concentration, which in turn affects the accuracy of identification and the efficiency of re-bonding. Due to the low efficiency of wire jump detection and repair, the retention time of the entire process increases, and production efficiency decreases accordingly, which ultimately affects the overall performance of the micro-assembly production line. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of low production efficiency mentioned in the above background technology, and to propose a system and method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture.

[0005] A first aspect of the present invention provides a method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture, the method comprising: Step S101, obtaining a detection image of a target product; Step S102, extracting features from the detection image to obtain a multi-scale feature map; Step S103, performing cross-scale channel fusion on the multi-scale feature map to obtain a fused feature map; Step S104, performing target detection according to the fused feature map to obtain a gold wire detection result; the detection result includes gold wire position information; Step S105, matching the detection result with the preset standard position, and marking the abnormal position of the gold wire bonding on the detection image; Wherein, step S102, step S103 and step S104 are implemented by a pre-trained gold wire recognition model; the gold wire recognition model is a model improved based on the RT-DETR neural network architecture.

[0006] Preferably, the RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder; Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: The preset Bottleneck_DySnake module is embedded at the end of the backbone network to obtain an improved feature extraction network; the calculation process of the Bottleneck_DySnake module includes: First, the input data is extracted through a 1×1 convolution layer, and then passed to the dynamic snake convolution layer for targeted feature extraction of slender objects. Then, a 1×1 convolution layer is used to reduce the number of channels and integrate information to obtain the target features. If the residual connection is enabled, the original input is added to the target feature and then output; otherwise, the target feature is directly output.

[0007] Preferably, compared with the original RT-DETR model, the gold wire recognition model further includes the following specific improvements: Applying a preset gated attention upsampling module GMAU to the upsampling process of the hybrid encoder; the GMAU includes a first upsampling branch, a second upsampling branch and a first attention calculation branch; The GMAU calculation process includes: The first attention calculation branch first performs global average pooling on the input feature map X1 to obtain the first global context information; then the first global context information is converted into a channel attention weight T1 through a convolution layer and a Hardsigmoid activation function; The first upsampling branch uses transposed convolution to improve the resolution of the feature map X1 to obtain the feature map U1; The second upsampling branch uses interpolation to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2; After the feature map U1 and the feature map U2 are concatenated, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 to output the upsampled feature map.

[0008] Preferably, compared with the original RT-DETR model, the gold wire recognition model further includes the following specific improvements: Applying a preset gated attention downsampling module GMAD to the downsampling process of the hybrid encoder; the GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch; The calculation process of the GMAD includes: The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into a channel attention weight T2 through a convolution layer and a Hardsigmoid activation function; The first downsampling branch uses a 3×3 convolution with a step size of 2 to perform feature dimension reduction and spatial compression on the feature map X2 to obtain a feature map D1; The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2; After the feature map D1 and the feature map D2 are concatenated, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 to output the downsampled feature map.

[0009] Preferably, compared with the original RT-DETR model, the gold wire recognition model further includes the following specific improvements: A preset cross-stage parallel dilation rotation equivariant convolution module CSP_RPDC is applied to the cross-scale fusion process of the hybrid encoder. Specifically, the RepC3 module in the hybrid encoder is replaced with the CSP_RPDC; the CSP_RPDC includes a parallel dilation convolution layer and a rotation equivariant convolution layer; The calculation process of CSP_RPDC includes: The parallel expansion convolution layer uses convolution kernels with different expansion rates to process the input feature map X3 in parallel, captures features of different receptive fields, and performs cross-scale feature interaction through channel splicing and 1×1 convolution layer to obtain the feature map F1; The rotational equivariant convolution layer performs convolution operations on the input feature map X3 in multiple directions to enhance the expression ability of the features and obtain the feature map F2; After concatenating the feature map F1 and the feature map F2, a 1×1 convolutional layer is used to perform feature fusion and output the fused feature map.

[0010] A second aspect of the present invention provides a system for automatically identifying abnormal points of gold wire bonding based on a neural network architecture, the system comprising: An image acquisition module, used to acquire a detection image of a target product; The target detection module is used to call the pre-trained gold wire recognition model, perform feature extraction on the detection image, and obtain a multi-scale feature map; perform cross-scale channel fusion on the multi-scale feature map to obtain a fused feature map; perform target detection based on the fused feature map to obtain a gold wire detection result; the gold wire recognition model is a model improved based on the RT-DETR neural network architecture; the detection result includes gold wire position information; The defect marking module is used to match the detection result with a preset standard position and mark the abnormal position of the gold wire bonding on the detection image.

[0011] Preferably, the RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder; Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: The preset dynamic snake bottleneck module Bottleneck_DySnake is embedded at the end of the backbone network to obtain an improved feature extraction network; the calculation process of the Bottleneck_DySnake includes: First, the input data is extracted through a 1×1 convolution layer, and then passed to the dynamic snake convolution layer for targeted feature extraction of slender objects. Then, a 1×1 convolution layer is used to reduce the number of channels and integrate information to obtain the target features. If the residual connection is enabled, the original input is added to the target feature and then output; otherwise, the target feature is directly output.

[0012] Preferably, compared with the original RT-DETR model, the gold wire recognition model further includes the following specific improvements: A gated attention upsampling module GMAU is constructed according to the gating mechanism and the attention mechanism, and is applied to the upsampling process of the hybrid encoder; the GMAU includes a first upsampling branch, a second upsampling branch, and a first attention calculation branch; The calculation process of the gated attention upsampling module includes: The first attention calculation branch first performs global average pooling on the input feature map X1 to obtain the first global context information; then the first global context information is converted into a channel attention weight T1 through a convolution layer and a Hardsigmoid activation function; The first upsampling branch uses transposed convolution to improve the resolution of the feature map X1 to obtain the feature map U1; The second upsampling branch uses interpolation to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2; After the feature map U1 and the feature map U2 are concatenated, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 to output the upsampled feature map.

[0013] Preferably, compared with the original RT-DETR model, the gold wire recognition model further includes the following specific improvements: A gated attention downsampling module GMAD is constructed according to the gating mechanism and the attention mechanism, and is applied to the downsampling process of the hybrid encoder; the GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch; The calculation process of the GMAD includes: The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into a channel attention weight T2 through a convolution layer and a Hardsigmoid activation function; The first downsampling branch uses a 3×3 convolution with a step size of 2 to perform feature dimension reduction and spatial compression on the feature map X2 to obtain a feature map D1; The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2; After the feature map D1 and the feature map D2 are concatenated, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 to output the downsampled feature map.

[0014] Preferably, compared with the original RT-DETR model, the gold wire recognition model further includes the following specific improvements: According to the dilated convolution and the rotational equivariant convolution, a cross-stage parallel dilated rotational equivariant convolution module CSP_RPDC is constructed, and the RepC3 module in the hybrid encoder is replaced with the CSP_RPDC; the CSP_RPDC includes a parallel dilated convolution layer and a rotational equivariant convolution layer; The calculation process of CSP_RPDC includes: The parallel expansion convolution layer uses convolution kernels with different expansion rates to process the input feature map X3 in parallel, captures features of different receptive fields, and performs cross-scale feature interaction through channel splicing and 1×1 convolution layer to obtain the feature map F1; The rotational equivariant convolution layer performs convolution operations on the input feature map X3 in multiple directions to enhance the expression ability of the features and obtain the feature map F2; After concatenating the feature map F1 and the feature map F2, a 1×1 convolutional layer is used to perform feature fusion and output the fused feature map.

[0015] Beneficial effects of the present invention: The present invention proposes an automatic identification method for abnormal points of gold wire bonding based on a neural network architecture, the method comprising: acquiring a detection image of a target product; calling a pre-trained gold wire recognition model to extract features from the detection image to obtain a multi-scale feature map; cross-scale channel fusion of the multi-scale feature map to obtain a fused feature map; performing target detection based on the fused feature map to obtain a gold wire detection result; the gold wire recognition model is an improved model based on the RT-DETR neural network architecture; the detection result includes gold wire position information; matching is performed based on the detection result and a preset standard position, and the abnormal position of the gold wire bonding is marked on the detection image.

[0016] The product images are processed through the improved RT-DETR model to achieve automatic gold wire recognition and defect marking, so that operators can quickly locate the defects based on the defect markings and guide the completion of the gold wire re-printing operation, reducing the time for defect identification and repair and improving production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture is provided for an embodiment of the present invention; Figure 2 A network architecture diagram of a gold wire recognition model is provided for an embodiment of the present invention; Figure 3 A structural schematic diagram of a dynamic serpentine bottleneck structure is provided for an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a gated attention upsampling module is provided for an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a cross-stage parallel expansion and rotation equivariant convolution module is provided for an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a gated attention downsampling module is provided for an embodiment of the present invention; Figure 7 A schematic diagram for comparing training results before and after model improvement is provided for an embodiment of the present invention; in: Figure 7(a) shows a schematic diagram of the mAP50-95 average precision mean training of the original RT-DETR model; Figure 7 (b) is a schematic diagram showing the mAP50-95 average precision mean training of the gold wire recognition model proposed in the present invention; Figure 8 Another schematic diagram for comparing training results before and after model improvement is provided for an embodiment of the present invention; in: Figure 8 (a) is a schematic diagram showing the gold wire confidence of the original RT-DETR model in densely packed and difficult-to-identify areas; Figure 8 (b) is a schematic diagram showing the gold wire confidence of the gold wire recognition model proposed by the present invention in a densely arranged and difficult to recognize area; Fig. 9 An architecture diagram of a gold wire bonding abnormal point automatic identification system based on a neural network architecture is provided for an embodiment of the present invention; Fig.10 A defect marking schematic diagram is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The embodiment of the present invention provides a method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture. Figure 1 , Figure 1 A flowchart of a method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture is provided in an embodiment of the present invention. The method comprises the following steps: S101, obtaining a detection image of a target product.

[0020] S102, extracting features from the detection image to obtain a multi-scale feature map.

[0021] S103, performing cross-scale channel fusion on the multi-scale feature maps to obtain a fused feature map.

[0022] S104, performing target detection based on the fused feature map to obtain a gold wire detection result.

[0023] S105, matching is performed according to the detection result and the preset standard position, and the abnormal position of the gold wire bonding is marked on the detection image.

[0024] Among them, the detection results include gold wire position information.

[0025] Based on the method for automatically identifying gold wire bonding abnormal points based on a neural network architecture provided by an embodiment of the present invention, product images are processed through an improved RT-DETR model to achieve automatic gold wire recognition and defect marking, so that operators can quickly locate the defects based on the defect markings and guide the completion of the gold wire re-insertion operation, thereby reducing the time for defect identification and repair and improving production efficiency.

[0026] In one embodiment, steps S102, S103 and S104 are implemented by a pre-trained golden wire recognition model; the golden wire recognition model is a model improved based on the RT-DETR neural network architecture. The RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder. The backbone network is used to extract feature representations of the input image; the hybrid encoder is used to convert multi-scale features into image feature sequences through intra-scale feature interaction and cross-scale feature fusion; the decoder is used to perform target detection and generate bounding boxes and confidence scores.

[0027] See also Figure 2 , Figure 2 A network architecture diagram of a gold wire recognition model is provided for an embodiment of the present invention. As shown in the figure, the backbone network and hybrid encoder are modified. Specifically: 1. Propose a self-developed Bottleneck_DySnake module.

[0028] Based on the residual bottleneck structure, the second layer of convolution is replaced by dynamic snake convolution to obtain the dynamic snake bottleneck module Bottleneck_DySnake. Figure 3 , Figure 3 A schematic diagram of a dynamic snake bottleneck structure provided by an embodiment of the present invention. The calculation process of the Bottleneck_DySnake module includes: firstly, feature extraction of the input data is performed through a 1×1 convolution layer, and then the feature extraction of the slender object is performed through the dynamic snake convolution layer, and then the number of channels is reduced through a 1×1 convolution layer and the information is integrated to obtain the target feature; if the residual connection is enabled (shortcut=true), the original input is added to the target feature and then output, otherwise the target feature is directly output.

[0029] Among them, the dynamic snake convolution layer contains a three-branch structure of a standard convolution Conv3×3 (followed by batch normalization BN and ReLU activation function), x-axis dynamic snake convolution DSConv_X (followed by group normalization GN and ReLU activation function) and y-axis dynamic snake convolution DSConv_Y (followed by group normalization GN and ReLU activation function). The outputs of the three branches are spliced ​​and input into the next layer.

[0030] In one implementation, the Bottleneck_DySnake module is embedded at the end of the backbone network to obtain an improved feature extraction network. Specifically, the improved feature extraction network consists of three ConvNormLayer modules, one MaxPool layer, three Bottleneck blocks modules and one Bottleneck_DySnake blocks module. Specifically: First, the network structure uses three ConvNormLayer modules. These three modules are responsible for preliminary feature extraction of the input image. Each ConvNormLayer module contains a convolution layer, a normalization layer, and a ReLU activation function. The convolution layer extracts local features through a 3×3 convolution kernel, the normalization layer helps to accelerate model training and improve generalization ability, and the activation function enhances the nonlinear expression ability of the model. These three modules gradually increase the number of channels from 32 channels to 64 channels, allowing the network to capture richer image features.

[0031] Next, the network introduces a MaxPool layer, which reduces the size of the feature map by half through a 3×3 pooling kernel with a step size of 2, providing a more compact feature representation for the subsequent deep network.

[0032] Subsequently, the network contains three Bottleneck blocks, which is an efficient feature extraction structure. Each Bottleneck blocks module consists of multiple BottleNeck units, and each unit achieves cross-channel feature fusion and dimensionality reduction through 1×1, 3×3, and 1×1 convolutional layers, effectively reducing the number of parameters and calculations. In one implementation, a feasible configuration is: the number of Bottleneck units in the first Bottleneck blocks module to the third Bottleneck blocks module is 3, 4, and 6, respectively.

[0033] Finally, in view of the thin and long characteristics of the gold wire, the network introduced the self-developed module Bottleneck_DySnakeblocks. In one implementation, the feasible configuration is: the number of Bottleneck_DySnake blocks contained in the Bottleneck_DySnake blocks is 3.

[0034] The advantages of the entire backbone network structure are mainly reflected in the following aspects: 1. Hierarchical feature extraction: From the shallow layer to the deep layer, the network gradually extracts the abstract features of the image, which is conducive to the subsequent task recognition. 2. Parameter efficiency: By using the Bottleneck structure and 1×1 convolution, the network reduces the number of parameters while ensuring the richness of features. 3. Multi-scale feature fusion: The MaxPool layer and multiple Bottleneck blocks modules realize multi-scale feature fusion, enabling the network to cope with targets of different scales. 4. Dynamic convolution: The introduction of dynamic snake convolution makes the network more adaptable to gold wire bonding scenarios.

[0035] 2. Propose a self-developed gated attention upsampling module (GMAU, Gate MechanismAttention Upsample). Figure 4 , Figure 4 A schematic diagram of the structure of a gated attention upsampling module provided in an embodiment of the present invention. The GMAU includes a first upsampling branch, a second upsampling branch, and a first attention calculation branch; The calculation process of GMAU includes: The first attention calculation branch first performs global average pooling (GAP) on the input feature map X1 to obtain the first global context information; then the first global context information is converted into the channel attention weight T1 through a convolution layer and a Hardsigmoid activation function; The first upsampling branch uses transposed convolution (ConvTranspose) to improve the resolution of feature map X1 and obtain feature map U1; The second upsampling branch uses interpolation for upsampling (Upsample) to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2; After concatenating the feature map U1 and the feature map U2, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3. Finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 and output the upsampled feature map.

[0036] In one implementation, the gated attention upsampling module is applied to the upsampling process of the hybrid encoder. The parallel upsampling branch can provide multiple feature extraction paths for the network, enrich the diversity of feature expression, and then combine the gate mechanism to select the features after sampling, strengthen the features of more meaningful and relevant gold wires, suppress redundant or irrelevant features, and improve the effectiveness of feature expression.

[0037] 3. Propose a self-developed cross-stage parallel dilation rotation equivariant convolution module (CSP_RPDC, CrossStage Partial_ Rotation-equivariant Parallel Dilated Convolution). See Figure 5 , Figure 5 A schematic diagram of the structure of a cross-stage parallel dilated rotation equivariant convolution module provided by an embodiment of the present invention. CSP_RPDC includes a parallel dilated convolution layer (PDC, ParallelDilatedConv) and a rotation equivariant convolution layer (RotC, Rotation-equivariant Convolution); The calculation process of CSP_RPDC includes: The parallel dilated convolution layer PDC uses dilated convolutions (DilatedConv) with different dilation rates to process the input feature map X3 in parallel, capture the features of different receptive fields, and concatenate the features of different scales according to the channel dimension. The 1×1 convolution layer performs cross-scale feature interaction to obtain the feature map F1. Specifically, three groups of convolution kernels with dilation rates of 1, 2, and 3 can be used to synchronously capture the local texture, medium-range topological structure, and global morphological features of the gold wire target.

[0038] The rotational equivariant convolution layer RotC performs convolution operations on the input feature map X3 in multiple directions to enhance the feature expression ability and obtain the feature map F2. When detecting gold wires, since there are gold wires with various rotation angles, RotC with rotation invariance is selected for identification. Specifically, the rotational convolution performs rotation operations on the input features in four directions of 0°, 90°, 180°, and 270°, and shares the convolution weights to extract direction-independent features, which effectively copes with the arbitrary angle distribution of gold wire targets in the image.

[0039] After concatenating the feature map F1 and the feature map F2, a 1×1 convolutional layer is used to perform feature fusion and output the fused feature map.

[0040] In one implementation, the cross-stage parallel dilation rotation equivariant convolution module CSP_RPDC is applied to the cross-scale fusion process of the hybrid encoder. Specifically, the RepC3 module in the hybrid encoder is replaced by CSP_RPDC. The advantages of the entire module are: first, it can perform multi-directional feature extraction. The RotC layer can identify the directional features in the image and improve the detection ability of gold wires that are sensitive to rotation or direction. Second, multi-scale feature fusion is performed. The PDC layer and the RotC layer work together to enable the module to adapt to gold wires of different scales and improve the accuracy of detection.

[0041] 4. Propose a self-developed gated attention downsampling module (GMAD, Gate Mechanism Attention Downsample). Figure 6 , Figure 6 A schematic diagram of the structure of a gated attention downsampling module provided in an embodiment of the present invention. GMAD includes a first downsampling branch, a second downsampling branch and a second attention calculation branch; The calculation process of GMAD includes: The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into the channel attention weight T2 through a convolution layer and a Hardsigmoid activation function; The first downsampling branch uses a 3×3 convolution with a step size of 2 to reduce the dimension and compress the feature map X2 to obtain the feature map D1; The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of feature map X2 to obtain feature map D2; After concatenating feature map D1 and feature map D2, they are multiplied channel by channel with the channel attention weight T2 to obtain feature map D3. Finally, a 1×1 convolutional layer is used to reorganize the cross-channel information of feature map D3 and output the downsampled feature map.

[0042] In one implementation, the gated attention downsampling module GMAD is applied to the downsampling process of the hybrid encoder. In the downsampling process, the global context information can be used to weight the features, thereby retaining more important information. This design helps to improve the robustness and generalization ability of the model in complex scenarios. By combining the gate mechanism with the downsampling operation, the module can effectively extract and fuse features to provide support for subsequent object detection tasks.

[0043] In one embodiment, the proposed modules are embedded in the RT-DETR model to obtain a gold thread recognition model. The backbone network, i.e., the improved feature extraction network, adopts a progressive feature extraction architecture to achieve multi-scale feature learning. The hybrid encoder performs feature enhancement through top-down and bottom-up bidirectional interactions, and the fused feature maps are output by the last three layers of CSP_RPDC modules respectively. The fused multi-scale feature map is fed into the RT-DETR Decoder detection decoder, which is responsible for generating the final detection result. In the decoder, the feature map is further processed to generate the position and bounding box prediction of the gold thread.

[0044] In one embodiment, the entire training process uses a back propagation algorithm to guide the update of network parameters by calculating loss functions such as classification loss, localization loss, and bounding box regression loss. This process is iterated continuously until the network reaches a predetermined performance indicator, thereby completing the model training.

[0045] The training process of the model is as follows: First, the data set needs to be collected and produced. This process builds a data set based on the 880 gold wire image samples collected, annotates the image samples (including gold wire targets) in the data set, and generates an annotation file corresponding to the image samples. Furthermore, the training data set is enhanced by color jitter, flipping, scaling and cropping, Gaussian blur, Gaussian noise injection, etc.

[0046] Then, the training set, test set, and validation set are divided in a ratio of 7:1:2. The training set is used to train the model, and the model adjusts its parameters through the data of the training set to minimize the loss function. The validation set is used to evaluate the performance of the model during the training process and guide the adjustment and selection of the model's hyperparameters. The test set is used to finally evaluate the performance of the model. It is used after the model is fully trained to help verify the performance of the model on unseen data.

[0047] During the training process, a dynamic label allocation strategy is adopted, and the matching degree between the predicted box and the real box is optimized through the Hungarian algorithm. At the same time, a composite loss function is designed: Focal Loss is used to solve the category imbalance problem in the classification task, and L1 loss and GIoU loss are used to jointly supervise the accuracy of the bounding box in the regression task. The direction consistency constraint loss is specially introduced for the slender characteristics of the gold wire. When the model updates the parameters through gradient back propagation, an adaptive learning rate scheduling strategy is adopted. In the early stage, the training of the low-level feature extraction network is focused, and in the middle and late stages, the coordinated optimization of high-level semantic features and the decoder attention mechanism is gradually strengthened, so that the model can finally have the ability to stably detect micron-level gold wires in complex industrial scenarios.

[0048] The gold wire bonding anomaly recognition system proposed in the present invention is highly robust. Even if the image is affected by factors such as rotation, scaling or lighting changes, the system can still relatively stably match key points and identify differences. It has strong robustness and can adapt to different image scenes.

[0049] For training results and comparison before and after improvement, see Figure 7 and Figure 8 . Figure 7 (a) shows the mAP50-95 mean average precision of the original RT-DETR model (the corresponding mAP50-95 mean average precision means: after 200 rounds of training, the intersection-over-union ratio threshold is 0.5 to 0.95, 10 mAP values ​​are obtained at an interval of 0.05, and then the average precision is obtained); Figure 7 (b) in the figure shows the mAP50-95 average precision of the gold wire recognition model proposed in the present invention. The mAP50-95 average precision of the gold wire stable recognition is improved from 0.574 to 0.671. Figure 8 (a) shows the gold wire confidence of the original RT-DETR model in densely packed and difficult-to-identify areas; Figure 8 (b) in the figure shows the gold wire confidence of the gold wire recognition model proposed in the present invention in the densely arranged and difficult to recognize area; after the improvement, the confidence of gold wire recognition is increased from 71% to 84%.

[0050] The embodiment of the present invention provides a system for automatically identifying abnormal points of gold wire bonding based on a neural network architecture. Fig. 9 , Fig. 9 The present invention provides an architecture diagram of a system for automatically identifying abnormal points of gold wire bonding based on a neural network architecture. The system includes: The image acquisition module is used to acquire the detection image of the target product.

[0051] The target detection module is used to call the pre-trained gold wire recognition model, extract features from the detection image, and obtain a multi-scale feature map; perform cross-scale channel fusion on the multi-scale feature map to obtain a fused feature map; perform target detection based on the fused feature map to obtain the gold wire detection result.

[0052] The defect marking module matches the inspection results with the preset standard positions and marks the abnormal positions of the gold wire bonds on the inspection images.

[0053] Among them, the gold thread recognition model is a model improved based on the RT-DETR neural network architecture; the detection results include the gold thread position information.

[0054] Based on the embodiment of the present invention, a neural network architecture-based automatic identification system for gold wire bonding abnormal points is provided. The product image is processed by an improved RT-DETR model to realize automatic gold wire identification and defect marking, so that the operator can quickly locate the defect according to the defect marking and guide the completion of the gold wire re-insertion operation, which reduces the time for defect identification and repair and improves production efficiency.

[0055] In one embodiment, the system hardware for applying the gold wire recognition model includes an electron microscope (with a photographing function) and a computer (meeting the requirements for running a gold wire defect recognition program), and the software includes an automatic gold wire defect recognition program.

[0056] The process of automatic identification of gold wire bonding abnormal points based on the neural network architecture is as follows: 1. During the batch production of RF MCM modules, the first piece completes all processes including gold wire bonding according to the process flow. At this time, the microwave PCB substrate sintering, surface mount component welding and bare chip carrier sintering have been completed. All bare chips and microwave PCBs have been installed in place. After manually confirming that the first piece of module is assembled correctly, the first piece of module is handed over to the inspection station for photography using an electron microscope and entered into the computer to establish a comparison benchmark as a standard image of a gold wire bonding module without wire jump defects. Then enter the batch production stage of RF MCM modules. After the module completes the fully automatic gold wire bonding, the batch production module is sent to the inspection station for photography using an electron microscope and entered into the computer.

[0057] 2. Import standard images of gold wire bonding modules without wire skipping defects, use the mature trained model to accurately identify these standard images, and obtain the standard position, that is, the detection frame position of the standard image.

[0058] 3. Detect and analyze the images of batch products. This process outputs the coordinates of all gold wires through the gold wire recognition model for the photos of batch products. Map the standard position to the image of the batch product. If the gold wire cannot be detected near the corresponding coordinates in the batch product image, for example, the coordinate deviation of the four corners of the detection frame reaches 20 pixels, it is determined that there is a wire jump error at this position in the batch product image. Mark the position where the wire jump error exists with a blue frame and label it "LOSS", such as Fig.10 As shown, accurate monitoring of the gold wire bonding quality and effective identification of defects are ensured, thereby guiding the staff to perform the gold wire re-bonding operation. The operator quickly identifies and locates the position of the label "LOSS" and guides the completion of the gold wire re-bonding operation. After completing the gold wire re-bonding operation, photos should be taken and identified again to confirm that all gold wire jump positions have completed the gold wire re-bonding operation.

[0059] It should be noted that, in this article, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0060] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture, characterized in that: The method comprises: Step S101, obtaining a detection image of a target product; Step S102, extracting features from the detection image to obtain a multi-scale feature map; Step S103, performing cross-scale channel fusion on the multi-scale feature map to obtain a fused feature map; Step S104, performing target detection according to the fused feature map to obtain a gold wire detection result; the detection result includes gold wire position information; Step S105, matching the detection result with the preset standard position, and marking the abnormal position of the gold wire bonding on the detection image; Wherein, step S102, step S103 and step S104 are implemented by a pre-trained gold wire recognition model; the gold wire recognition model is a model improved based on the RT-DETR neural network architecture.

2. The method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture according to claim 1, characterized in that: The RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder; Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: The preset dynamic snake bottleneck module Bottleneck_DySnake is embedded at the end of the backbone network to obtain an improved feature extraction network; the calculation process of the Bottleneck_DySnake includes: First, the input data is extracted through a 1×1 convolution layer, and then passed to the dynamic snake convolution layer for targeted feature extraction of slender objects. Then, a 1×1 convolution layer is used to reduce the number of channels and integrate information to obtain the target features. If the residual connection is enabled, the original input is added to the target feature and then output; otherwise, the target feature is directly output.

3. The method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture according to claim 2, characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: Applying a preset gated attention upsampling module GMAU to the upsampling process of the hybrid encoder; the GMAU includes a first upsampling branch, a second upsampling branch and a first attention calculation branch; The GMAU calculation process includes: The first attention calculation branch first performs global average pooling on the input feature map X1 to obtain the first global context information; then the first global context information is converted into a channel attention weight T1 through a convolution layer and a Hardsigmoid activation function; The first upsampling branch uses transposed convolution to improve the resolution of the feature map X1 to obtain the feature map U1; The second upsampling branch uses interpolation to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2; After the feature map U1 and the feature map U2 are concatenated, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 to output the upsampled feature map.

4. The method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture according to claim 3, characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: Applying a preset gated attention downsampling module GMAD to the downsampling process of the hybrid encoder; the GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch; The calculation process of the GMAD includes: The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into a channel attention weight T2 through a convolution layer and a Hardsigmoid activation function; The first downsampling branch uses a 3×3 convolution with a step size of 2 to perform feature dimension reduction and spatial compression on the feature map X2 to obtain a feature map D1; The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2; After the feature map D1 and the feature map D2 are concatenated, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 to output the downsampled feature map.

5. The method for automatically identifying abnormal points of gold wire bonding based on a neural network architecture according to claim 4, characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: A preset cross-stage parallel dilation rotation equivariant convolution module CSP_RPDC is applied to the cross-scale fusion process of the hybrid encoder. Specifically, the RepC3 module in the hybrid encoder is replaced with the CSP_RPDC; the CSP_RPDC includes a parallel dilation convolution layer and a rotation equivariant convolution layer; The calculation process of CSP_RPDC includes: The parallel expansion convolution layer uses convolution kernels with different expansion rates to process the input feature map X3 in parallel, captures features of different receptive fields, and performs cross-scale feature interaction through channel splicing and 1×1 convolution layer to obtain the feature map F1; The rotational equivariant convolution layer performs convolution operations on the input feature map X3 in multiple directions to enhance the expression ability of the features and obtain the feature map F2; After concatenating the feature map F1 and the feature map F2, a 1×1 convolutional layer is used to perform feature fusion and output the fused feature map.

6. A gold wire bonding abnormal point automatic identification system based on a neural network architecture, characterized in that: The system comprises: An image acquisition module, used to acquire a detection image of a target product; The target detection module is used to call the pre-trained gold wire recognition model, perform feature extraction on the detection image, and obtain a multi-scale feature map; perform cross-scale channel fusion on the multi-scale feature map to obtain a fused feature map; perform target detection based on the fused feature map to obtain a gold wire detection result; the gold wire recognition model is a model improved based on the RT-DETR neural network architecture; the detection result includes gold wire position information; The post-processing module performs matching according to the detection result and the preset standard position, and marks the abnormal position of the gold wire bonding on the detection image.

7. The system for automatically identifying abnormal gold wire bonding points based on a neural network architecture according to claim 6, characterized in that: The RT-DETR neural network architecture includes a backbone network, a hybrid encoder and a decoder; Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: The preset dynamic snake bottleneck module Bottleneck_DySnake is embedded at the end of the backbone network to obtain an improved feature extraction network; the calculation process of the Bottleneck_DySnake includes: First, the input data is extracted through a 1×1 convolution layer, and then passed to the dynamic snake convolution layer for targeted feature extraction of slender objects. Then, a 1×1 convolution layer is used to reduce the number of channels and integrate information to obtain the target features. If the residual connection is enabled, the original input is added to the target feature and then output; otherwise, the target feature is directly output.

8. The automatic identification system for gold wire bonding abnormal points based on a neural network architecture according to claim 7 is characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: A gated attention upsampling module GMAU is constructed according to the gating mechanism and the attention mechanism, and is applied to the upsampling process of the hybrid encoder; the GMAU includes a first upsampling branch, a second upsampling branch, and a first attention calculation branch; The calculation process of the gated attention upsampling module includes: The first attention calculation branch first performs global average pooling on the input feature map X1 to obtain the first global context information; then the first global context information is converted into a channel attention weight T1 through a convolution layer and a Hardsigmoid activation function; The first upsampling branch uses transposed convolution to improve the resolution of the feature map X1 to obtain the feature map U1; The second upsampling branch uses interpolation to improve the resolution of the feature map X1, and then performs feature mapping through 1×1 convolution to obtain the feature map U2; After the feature map U1 and the feature map U2 are concatenated, they are multiplied channel by channel with the channel attention weight T1 to obtain the feature map U3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map U3 to output the upsampled feature map.

9. The automatic identification system for gold wire bonding abnormal points based on neural network architecture according to claim 8 is characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: A gated attention downsampling module GMAD is constructed according to the gating mechanism and the attention mechanism, and is applied to the downsampling process of the hybrid encoder; the GMAD includes a first downsampling branch, a second downsampling branch, and a second attention calculation branch; The calculation process of the GMAD includes: The second attention calculation branch first performs global average pooling on the input feature map X2 to obtain the second global context information; then the second global context information is converted into a channel attention weight T2 through a convolution layer and a Hardsigmoid activation function; The first downsampling branch uses a 3×3 convolution with a step size of 2 to perform feature dimension reduction and spatial compression on the feature map X2 to obtain a feature map D1; The second downsampling branch uses maximum pooling combined with 1×1 convolution to reduce the resolution of the feature map X2 to obtain the feature map D2; After the feature map D1 and the feature map D2 are concatenated, they are multiplied channel by channel with the channel attention weight T2 to obtain the feature map D3; finally, a 1×1 convolution layer is used to reorganize the cross-channel information of the feature map D3 to output the downsampled feature map.

10. The system for automatically identifying abnormal points of gold wire bonding based on a neural network architecture according to claim 9, characterized in that: Compared with the original RT-DETR model, the gold wire recognition model has the following specific improvements: According to the dilated convolution and the rotational equivariant convolution, a cross-stage parallel dilated rotational equivariant convolution module CSP_RPDC is constructed, and the RepC3 module in the hybrid encoder is replaced with the CSP_RPDC; the CSP_RPDC includes a parallel dilated convolution layer and a rotational equivariant convolution layer; The calculation process of CSP_RPDC includes: The parallel expansion convolution layer uses convolution kernels with different expansion rates to process the input feature map X3 in parallel, captures features of different receptive fields, and performs cross-scale feature interaction through channel splicing and 1×1 convolution layer to obtain the feature map F1; The rotational equivariant convolution layer performs convolution operations on the input feature map X3 in multiple directions to enhance the expression ability of the features and obtain the feature map F2; After concatenating the feature map F1 and the feature map F2, a 1×1 convolutional layer is used to perform feature fusion and output the fused feature map.

Images