A hole segmentation method for electronic component X-ray images based on aggregated convolution and self-attention features

Abstract

This method discloses a method for segmenting holes in X-ray images of electronic components based on aggregated convolution and self-attention features. First, data augmentation is performed on the component dataset to expand the data sample size and avoid overfitting. Secondly, the data is input into the model for segmentation. The model is designed as a U-shaped Encoder-Decoder structure. The backbone network with mixed branches of CNN and Transformer is introduced as the Encoder module to extract features, aggregating the local and global features of the image. The Decoder structure is designed to upsample the output of the Encoder module, and skip connections are used in each layer of upsampling. The downsampled features are fused to make the subsequent upsampled features more refined, the image restoration more accurate, and the segmentation accuracy higher. Finally, the LovaszSoftmax loss function is used to guide the training to obtain the segmentation model. This method can be used for the task of segmenting surface hole defects of electronic components, and has the advantages of low cost, real-time detection, and high accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation in computer vision and is a method for classifying each pixel in an image using a convolutional neural network. Background Art

[0002] With rising domestic labor costs, expanding production capacity demands, support from relevant policies, and numerous industrial developments, new methods such as deep learning-based intelligent product defect detection are rapidly emerging. Deep learning-based intelligent product defect detection will also be increasingly applied in modern manufacturing. Replacing manual labor with deep learning-based product surface defect detection algorithms will also become an inevitable trend.

[0003] With the popularity of deep learning, semantic segmentation research has also made significant progress, achieving excellent segmentation results on general datasets. However, when classic semantic segmentation algorithms are directly applied to surface defect detection tasks, the segmentation results and algorithm real-time performance are not ideal. These problems are mainly attributed to the following aspects: Although increasing the depth of deep convolutional networks makes it easier to extract abstract features, the continuous downsampling operation will lead to a decrease in image resolution and the loss of more precise location details. This is a significant drawback for semantic segmentation and other low-level vision tasks that require this precise location information. At the same time, increasing the depth will increase the number of model parameters and make the calculations more complex, leading to problems such as overfitting during model training and low real-time segmentation rates. Summary of the Invention

[0004] At present, common semantic segmentation methods include FCN networks, U-Net networks, DeepLab series algorithms, etc. These methods have achieved good segmentation effects on general data sets. However, when migrating the algorithms to the segmentation of surface defects of components, there are still problems such as low segmentation accuracy, large number of model parameters and large amount of calculation, which makes the algorithm model difficult to use in actual engineering projects. To this end, the present invention designs a void segmentation method for X-ray images of electronic components based on aggregated convolution and self-attention features, which takes into account the accuracy and real-time requirements of model segmentation and has good segmentation effects on UB package type components; in response to the problems of inaccurate segmentation and low segmentation rate of the above-mentioned algorithm, the present invention designs a void segmentation method for X-ray images of electronic components based on aggregated convolution and self-attention features, which achieves high-precision and high-efficiency defect segmentation.

[0005] The present invention adopts the following technical solutions

[0006] A method for hole segmentation in X-ray images of electronic components based on aggregated convolution and self-attention features. The specific process of hole segmentation in electronic component images is as follows: first, the X-ray image of the electronic component is preprocessed and data enhanced, and then the image is fed as input into the proposed network. In the model, the image first passes through the Encoder module, where local and global features of the image are extracted. The local and global features are then fused through the FCU module to retain the image features to the greatest extent possible. Secondly, the features fused by the Feature Coupling Unit (FCU) module are fed into the Bridge module for a second feature fusion. Finally, the Decoder module performs layer-by-layer upsampling, and skip connections are used to aggregate feature information of the same scale, forming a highly flexible feature fusion scheme to restore the spatial detail information of the original image and obtain the final segmentation result.

[0007] (1) Data preprocessing: This paper uses collected images of electronic components with UB packages as a dataset. To facilitate training and prediction, the collected images must be processed. First, the blank background of the image is cropped. Second, the non-UB package components in the image are cropped to obtain images containing only complete UB package electronic components. Finally, all images are batch processed using bilinear interpolation and resized to 224×192.

[0008] (2) Data augmentation: Due to the insufficient number of defect samples collected, data augmentation was performed on the UB package component dataset to improve the generalization ability of the model. Data augmentation methods used before training included normalization, random cropping, and random scaling. The random scaling factor was set in the interval (0, 2) with an interval of 0.25.

[0009] (3) Model setup and training: The processed UB package type dataset images are used as network input for training. The network model is a U-shaped encoder-decoder structure. The encoder module first extracts the global and local features of the input through the hybrid branch of CNN and Transformer, and then fuses the hybrid features through the Feature Coupling Unit (FCU) module. The fused features output by the encoder module are fused twice through the 3x3 convolution bridge module. After the feature fusion is completed, it is used as input for image restoration through the decoder module, including 4 steps of upsampling operation. Each upsampling step corresponds to 4 dual-branch downsampling stages. In the process of upsampling stage by stage, the corresponding stage features are spliced ​​through jump connections to fuse more feature information, and finally the image is restored to the original resolution to obtain the segmentation result.

[0010] During training, no pre-trained models are required. The model is trained using the LovaszSoftmax loss function based on IoU to achieve multi-class segmentation. The SGD optimizer with momentum is used during training, with an initial learning rate of 0.003 and 300 epochs.

[0011] (4) Model prediction: After the model is trained, the model weight file is saved for subsequent prediction. Model prediction requires first loading the trained model parameters and then inputting images of UB-type components of any size as test data. This allows the location of void defects in the image to be obtained end-to-end. At this stage, only the trained model parameters need to be loaded; data augmentation is not used in this step.

[0012] The evaluation metrics used were the mean intersection over union (mIoU), one of the most commonly used metrics for segmentation benchmark datasets, and the segmentation rate evaluation metric, FPS. The performance of the entire algorithm was evaluated on a UB packaged dataset. The proposed method achieved competitive results, achieving higher segmentation accuracy and rate than current classic semantic segmentation algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The figure is a schematic diagram of the overall process of the method proposed in the present invention.

[0014] Figure 2 Schematic diagram of the model framework proposed in this invention.

[0015] Figure 3 This is a schematic diagram of the network structure proposed by the present invention. DETAILED DESCRIPTION

[0016] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings:

[0017] A hole segmentation method for electronic component X-ray images based on aggregated convolution and self-attention features. Figure 1 As shown in the figure, the overall process involves preprocessing and data augmentation (applied only during the training phase) of X-ray images of electronic components. The processed images of UB packages are then fed into the network for training. In the entire model, the input image is first downsampled by the Encoder module to extract image features. The Bridge feature fusion module then fuses the mixed features extracted by the Encoder module for a second time. The fused features are then fed into the Decoder module for upsampling. Skip connections are used to fuse features from different levels, and classification and regression are performed to produce the segmentation results.

[0018] The specific algorithm is as follows:

[0019] (1) Data preprocessing: First, the blank background of the image is cropped. Second, the non-UB package type component parts in the image are cropped to obtain an image containing only complete UB package type electronic components. Finally, all image sizes are batch processed using the bilinear interpolation method. The entire model involves a four-stage hybrid branch network in the encoder module, which performs four downsampling operations and three upsampling fusion features in the decoder module to ensure that each operation is divisible by 32, and the size is adjusted to 224×192.

[0020] (2) Data enhancement: Common segmentation algorithms require a large amount of data for training to improve algorithm robustness. Since the amount of component defect sample data collected in actual scenarios is relatively small and the labeling cost is relatively high, data enhancement steps are performed on the existing data during training to expand the data volume. The data enhancement methods used in the training process for the pre-processed UB type electronic component dataset include normalization, random cropping, random scaling, etc. Among them, the random scaling factor is set in the interval of 0.25 in the range of (0, 2).

[0021] (3) Model setup and training

[0022] The entire model is designed as a U-shaped Encoder-Decoder structure with reference to the U-Net structure, such as Figure 2 The encoder part consists of a trunk module, two branches, and a Feature Coupling Unit (FCU) module that bridges the two branches.

[0023] Among them, the stem module is used to extract the initial local features. It consists of a 7×7 convolution with a stride of 2 and a 3×3 maximum pooling layer with a stride of 2 to downsample the model input.

[0024] The dual-branch structure refers to the CNN-Transformer Block, and the entire backbone consists of 12 blocks. The CNN-Transformer Block contains a CNN branch and a Transformer branch. The branches interact and fuse feature information through an aggregation module to maximize the preservation of local and global features. The CNN branch adopts a feature pyramid structure, where the resolution of the feature map decreases with increasing network depth, while the number of channels increases. The entire branch can be divided into four stages, each consisting of multiple convolutional blocks, each containing n bottlenecks. The convolution design in the bottlenecks follows the block style of ResNet, including 1×1 convolution, 3×3 spatial convolution, 1×1 convolution, and residual connections between the input and output. The Transformer branch primarily consists of transformer modules, each of which consists of a multi-head self-attention module and an MLP block. A LayerNorm layer is applied before each layer, and residual connections are designed between the attention and MLP layers.

[0025] In the dual-branch architecture, the Transformer branch feeds global context back to the convolutional module, strengthening the CNN branch's global perception. The CNN branch gradually feeds local features back into the patch embedding, enriching the Transformer branch's local details. The CNN and Transformer branches extract features in parallel, and the FCU module fuses the convolutional features and patch embeddings, accurately embedding the local features into the global representation.

[0026] The FCU module aggregates features between branches, fusing local features from the CNN branch with global features from the Transformer branch, interactively coupling local and global features. The CNN branch outputs feature maps with dimensions of C×H×W, while the patch embeddings have a shape of (K+1)×E. The feature dimensions of the CNN and Transformer branches are inconsistent. Therefore, when the CNN branch features are mapped to the Transformer branch, the feature map first undergoes a 1×1 convolution to align the number of channels of the patch embedding. Then, the spatial dimensions are aligned using a downsampling module, and the feature values ​​are aligned using LayerNorm and BatchNorm before being added to the patch embedding. When the Transformer branch features are mapped to the CNN branch, the patch embedding is upsampled to align the spatial scale. Then, the channel dimensions are aligned with the CNN branch feature map using a 1×1 convolution. The feature values ​​are aligned using LayerNorm and BatchNorm before being added to the CNN branch feature map. Finally, the FCU module is inserted into each block to eliminate the semantic differences between the CNN branch and the Transformer branch in a continuous interactive manner, greatly improving the global perception ability of local features and the local details of the global representation.

[0027] The Bridge module takes the mixed branch features of the Encoder module as input and performs secondary feature fusion, which makes the details finer and the restoration more accurate. It also reduces the dimension of the features, making it easier for the features to be restored by the Decoder module.

[0028] The decoder module upsamples features using interpolation, producing the segmentation result after four upsampling operations. Before upsampling each stage, detailed features of the same resolution as the backbone network are concatenated using skip connections. After the features undergo a 3×3 convolutional layer and interpolation upsampling, the predicted results are fed into the input image.

[0029] (4) Model prediction and evaluation

[0030] The training process uses the LovaszSoftmax loss function to guide training, and the weight file is saved after training. During model prediction evaluation, the weight file is loaded to obtain the model parameters. This allows end-to-end prediction of an image of a component in a UB package of any size and outputs the segmentation result. The model segmentation accuracy is evaluated using the mean intersection over union (mIoU) and the segmentation rate is evaluated using the FPS metric. The algorithm's prediction performance was evaluated on a dataset of electronic components in the UB package. Compared to the classic DeepLabV3+ method, the improved algorithm shows significant improvements. The comparison results are shown in Table 1.

[0031] Table 1 Comparison of prediction performance of the method proposed in this invention

[0032]

[0033] As shown in Table 1, on the UB package type electronic component dataset, the algorithm of the present invention improved by 0.6% compared with the DeepLabV3+ algorithm, and the real-time segmentation rate was increased by nearly 5 times. The experimental results prove that the algorithm of the present invention is effective and can segment UB package type electronic components more accurately and faster.

Claims

1. A hole segmentation method for electronic component X-ray images based on aggregated convolution and self-attention features, characterized by: include: The electronic component X-ray image hole segmentation method is implemented based on the electronic component X-ray image hole segmentation model. The electronic component X-ray image hole segmentation model is a U-shaped encoder-decoder structure. The front part of the electronic component X-ray image hole segmentation model uses a hybrid branch of CNN and Transformer as the encoder module to extract features, and aggregates local and global features of the image through the FCU module to retain the image semantic information to the greatest extent. The decoder module uses the encoder module output as input after completing feature fusion through the bridge module. In each layer of upsampling, jump connections are used to splice feature information of the same resolution in the encoder. The fused features can ensure the hole detail information of the original electronic component X-ray image, thereby obtaining the final segmentation result. The Bridge module takes the mixed branch features of the Encoder module as input and performs secondary feature fusion, which makes the details finer and the restoration more accurate. At the same time, it reduces the dimension of the features to facilitate feature restoration through the Decoder module. The Decoder module is an important step in restoring the downsampled features to the original image. The overall process is to use the output features of the Bridge module as the module input, and perform four upsamplings by interpolation. Each upsampling process corresponds to the four downsampling stages in the Encoder module. Before the upsampling operation, the skip connection method is used to splice the detail features of the same resolution of the backbone network. The spliced ​​features are then passed through 3 3. Convolution performs dimensionality reduction and upsampling operations, and finally outputs the segmentation effect map.

2. The method for hole segmentation in electronic component X-ray images based on aggregated convolution and self-attention features according to claim 1, characterized in that: The Encoder module consists of a head backbone module, a dual-branch module for mixed features, and an FCU module for bridging mixed features. The head backbone module processes the input image separately and then inputs the features into the dual-branch module. The dual branches are designed to focus on extracting local and global features of the image respectively, including CNN branches and Transformer branches. This module uses convolution operations and self-attention mechanisms to enhance representation learning. The features of each branch will be fused with local features and global representations at different resolutions through the FCU feature coupling module in an interactive splicing manner between branches, thereby retaining the semantic information of the image to the maximum extent.

Images