A flexible circuit board defect detection method based on Transformer
Through the Transformer-based detection method, multi-scale feature extraction and cross-scale fusion technology are used to solve the problems of diverse scales and complex feature processing in flexible circuit board defect detection, achieving more efficient and accurate defect detection.
Patent Information
- Application Number
- CN202411347084.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-26
AI Technical Summary
The existing flexible circuit board defect detection methods have high subjectivity, low efficiency, poor reliability, high labor intensity, difficulty in dealing with complex defects, and high requirements for imaging conditions, so they cannot effectively deal with defects in diversity scales.
Using a Transformer-based detection method, multi-scale features are extracted through Swin Transformer and feature pyramid network, combined with the cross-scale coordinated multi-head self-attention mechanism and channel attention coordination module, cross-scale feature fusion is performed, and final detection is performed using Mask RCNN.
It improves the accuracy and robustness of defect detection, can better process multi-scale information, and improves adaptability and performance in complex scenarios.
Smart Images

Figure CN118887209B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of defect recognition, and in particular to a flexible circuit board defect detection method based on Transformer. Background Art
[0002] Flexible Printed Circuit (FPC) is a circuit board made of flexible substrate, usually made of thin polyester or polyimide materials. FPC has excellent flexibility and bendability, and can adapt to complex curved surfaces and narrow spaces. Therefore, it is widely used in fields such as mobile devices, medical devices, and automotive electronics.
[0003] Due to the diversity of materials, processes and equipment used in the FPC manufacturing process, as well as the various types of defects that occur at different production stages, there are defects of various sizes on the FPC. Since the defect size of the FPC is relatively small, the defect may show different characteristics at different scales.
[0004] At present, there are two main methods for FPC defect detection: manual visual inspection and traditional machine vision inspection. However, manual visual inspection has many shortcomings, including high subjectivity, low efficiency, poor reliability, and high labor intensity. Traditional machine vision inspection is mainly based on the color and shape characteristics of defects, using image processing and traditional machine learning methods. Although it is effective to a certain extent, it is difficult to handle complex defect features and has high requirements for imaging conditions. Therefore, more efficient and accurate FPC defect detection methods are urgently needed to cope with increasingly complex production needs. Summary of the invention
[0005] In order to solve the problems of high subjectivity, low efficiency, poor reliability, high labor intensity, difficulty in processing complex defect features, and high requirements on imaging conditions in existing FPC defect detection methods, the present invention provides a flexible circuit board defect detection method based on Transformer, which obtains multi-scale defect feature information and learns the feature expression of visual elements of different scales to effectively characterize the diverse scales of FPC defects and further improve the accuracy of defect detection.
[0006] The technical solution adopted by the present invention is:
[0007] A flexible circuit board defect detection method based on Transformer, comprising the following steps:
[0008] Step S1: Import the FPC image data into the encoder for PFC defect recognition, and the encoder encodes the pixel information of the image, extracts the image features, and outputs the FPC image feature data; the FPC image feature data includes the channel dimension information collaborative interaction features of scale 1, scale 2 and scale 3, which are respectively , and ;
[0009] in, is the cooperative interaction feature of channel dimension information at scale one;
[0010] It is the collaborative interaction feature of channel dimension information of scale 2;
[0011] It is the collaborative interaction feature of channel dimension information of scale three;
[0012] When constructing the encoder, Swin Transformer is selected as the backbone network, and FPC defect recognition is performed through four stages;
[0013] Step S2: The , and , and the fourth stage feature Scale_4 obtained by the encoder when performing PFC defect recognition is imported into the feature pyramid network FPN, and the feature pyramid network FPN is integrated to unify the features extracted at different scales and output the features of five stages, respectively. , , , and ;
[0014] in, The first stage features output by the feature pyramid network FPN;
[0015] The second stage features output by the feature pyramid network FPN;
[0016] The third stage features output by the feature pyramid network FPN;
[0017] The fourth stage features output by the feature pyramid network FPN;
[0018] The fifth stage feature output by the feature pyramid network FPN;
[0019] Step S3: The , , , and The cross-scale adaptive feature fusion module CSAF is introduced. The cross-scale adaptive feature fusion module CSAF performs cross-scale fusion of features in the spatial dimension based on the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, and outputs the cross-scale fused FPC image data F csaf ;
[0020] Step S4: The cross-scale fused FPC image data F obtained in step S3 is csaf The decoder is imported and fused by the decoder to obtain the final FPC image detection feature map, thereby completing the detection and identification of FPC defects.
[0021] Furthermore, in step S1, when the encoder Encoder performs image feature extraction, the following steps are specifically included:
[0022] Step S11: importing the FPC image data into the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA fuses the key features of the FPC image at scales 1, 2, and 3 to obtain the scale 1, scale 2, and scale 3 information collaborative interaction features, which are respectively , and ; Among them, ici represents information collaborative interaction;
[0023] Step S12: The , and The cross-scale channel attention collaborative module CS-CAC is introduced. The cross-scale channel attention collaborative module CS-CAC combines the channel attention mechanism and cross-scale information transfer to obtain the channel dimension information collaborative interaction features of scale one, scale two and scale three, respectively. , and .
[0024] Furthermore, the cross-scale adaptive feature fusion module CSAF performs the following steps when performing cross-scale fusion, specifically:
[0025] Step S31: for the data obtained in step S2 , , , and , the adjacent i-th scale and j-th scale features and The feature uses 1×1 convolution to adjust the number of channels, and adjusts the features of another layer to the same size as the target feature by upsampling or downsampling. Then, two learnable adaptive weights are introduced in the feature fusion module ASFF-S. and , and then processed by the Sigmoid function, it is multiplied with the features of the two scales after resizing to obtain two adaptively adjusted features, which are and , and output;
[0026] Step S32: for the step S31 obtained and The cross-scale adaptive feature fusion module CSAF uses deep separable convolution to calculate self-attention in the spatial dimension, and introduces implicit position coding to implicitly encode the position information. Then, the feature information after implicit position coding is residually connected with the original feature to obtain two features, namely and , and output; where pse represents position and scale encoding;
[0027] Step S33: for the step S32 obtained and In the spatial dimension, cross-scale information collaborative interaction is carried out, and a spatial reduction module is introduced. A normalization layer is used to obtain two features after spatial information collaborative interaction, which are and ;
[0028] Step S34: Use a 1×1 convolutional layer to adjust the obtained in step S33 and The dimension of the channel is fused into a single feature; at the same time, the local feature F of the original feature is obtained through the depth-separable convolution local , and the global feature F global Perform residual connection fusion; finally, perform normalization through the Batch Normalization layer to obtain the cross-scale fused FPC image data F csaf .
[0029] Furthermore, the decoder selects Mask RCNN as a decoder.
[0030] Furthermore, the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA specifically performs the following steps when fusing the key features of the FPC image at scales 1, 2, and 3:
[0031] Step S111: import the FPC image data and extract scale 1, scale 2 and scale 3 features respectively, , and ;
[0032] in, Represents the height, width and number of color channels of scale-one feature in three-dimensional space;
[0033] Represents the height, width, and number of color channels of scale 2 features in three-dimensional space;
[0034] Represents the height, width and number of color channels of the three scale features in three-dimensional space;
[0035] Step S112: The , and The cross-scale channel attention collaborative module CS-CAC based on the cross-scale collaborative multi-head self-attention mechanism is respectively imported for sequence conversion, and then the projection matrices of scales 1, 2 and 3 are respectively used. , and , calculate the Q, K and V vectors of scale one, scale two and scale three respectively, , and , and output;
[0036] in, represents a three-dimensional matrix,
[0037] Q represents the query vector, which is used to determine the focus in the attention calculation;
[0038] By interacting with K, it decides what information to extract from V;
[0039] K is the key vector associated with the input, which is compared with Q to determine the importance of different parts of the input;
[0040] V is the actual value vector, which contains the specific information of the input. According to the interaction results of Q and K, it is determined which information in V should be paid attention to and extracted;
[0041] is the query matrix of scale one, which is used to project the input data into the query space of scale one to generate the query vector;
[0042] is the Key matrix of scale one, which is used to project the input data of scale one into the key space and generate the key vector;
[0043] It is the Value matrix of scale one, which is used to project the input data of scale one into the value space and generate the value vector;
[0044] is the query matrix of scale 2, which is used to project the input data of scale 2 into the query space and generate the query vector;
[0045] is the Key matrix of scale 2, which is used to project the input data of scale 2 into the key space and generate the key vector;
[0046] is the Value matrix of scale 2, which is used to project the input data of scale 2 into the value space and generate a value vector;
[0047] is the query matrix of scale 3, which is used to project the input data of scale 3 into the query space and generate the query vector;
[0048] is the Key matrix of scale three, which is used to project the input data of scale three into the key space and generate the key vector;
[0049] It is the Value matrix of scale three, which is used to project the input data into the value space of scale three and generate the value vector;
[0050] Step S113: Use the concatenate operation to connect the Q vectors of scale 1, scale 2, and scale 3 outputted from step S112, and use a 1×1 convolution to perform channel adjustment to obtain the fused vector Q Fusion ; Using the fusion vector Q Fusion Perform matrix multiplication with the transpose of the K vector of scale 1, scale 2 and scale 3 outputted in step S112, and calculate the self-attention matrices of scale 1, scale 2 and scale 3, which are respectively , and , and output;
[0051] Step S114: The self-attention matrices of scale 1, scale 2, and scale 3 obtained in step S113 are , and Multiply the V vectors of their respective scales to obtain collaborative features; a fully connected layer is used to adjust the dimension and map the output features of scales 1, 2, and 3 to Get in space , and .
[0052] Furthermore, the cross-scale channel attention coordination module CS-CAC specifically performs steps S121 to S125 when performing cross-scale information coordination and complementation:
[0053] Step S121: The step S11 obtained , and Perform in-depth pooling processing and output pooling results of scale 1, scale 2, and scale 3;
[0054] Step S122: The pooling results of scale 1, scale 2 and scale 3 obtained in step S121 are merged in the second dimension to form multi-scale pooling features of scale 1, scale 2 and scale 3, which are respectively , and , and output; where the subscript mspf represents that the feature belongs to the multi-scale pooling feature;
[0055] Step S123: Use a fully connected layer to transform the , and They are mapped to a dimension C1 higher than 2L, and three serialized features are obtained, which are , and , and output;
[0056] Step S124: obtained in step S123, , and Add scale type encoding; input the features of scales 1, 2, and 3 into N Transformer modules, and use the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA to realize cross-scale collaborative interaction of channel information. The features obtained are , and , and output; where cici represents the collaborative interaction of channel information;
[0057] Step S125: Use a multilayer perceptron and a Sigmoid activation function to calculate the value obtained in step S124 , and , we get the channel attention vectors of scale one, scale two, and scale three, respectively , and ; Then, the original features , and Multiply the channel attention vector by channel to obtain the channel dimension information collaborative interaction features of scale one, scale two, and scale three, respectively. , and , and output.
[0058] Furthermore, the step S121 is for , and In-depth pooling processing is performed by applying multi-scale average pooling and multi-scale maximum pooling techniques to adapt to the features of these different scales.
[0059] Furthermore, the step S121 is for , and The specific process of deep pooling is as follows:
[0060] Set four pooling scales: M1×N1, M2×N2, M3×N3, and M4×N4;
[0061] Then the pooled features Flattened into a one-dimensional feature vector , where M and N represent the sizes of different pooling factors;
[0062] The flattened vectors are merged in the second dimension to form pooling results of scale one, scale two, and scale three, respectively. , , , , and , where L=M1N1+M2N2+M3N3+M4N4, and the subscripts mp and ap represent maximum pooling and average pooling.
[0063] The beneficial effects of the present invention are:
[0064] The present invention acquires multi-scale defect feature information and learns the feature expression of visual elements of different scales to effectively characterize the diverse scales of FPC defects and further improve the accuracy of defect detection. First, after inputting the image data, the image is feature extracted, and the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA is used to promote the effective interaction between features of different scales and enhance the model's perception of multi-scale information. Secondly, a cross-scale channel attention collaborative module CS-CAC is designed to extract feature information of three scales at each stage, and the channel dimension performs cross-scale information collaborative complementation to better adapt to image features at different scales. Then, the feature pyramid network FPN is used as the "neck" of the architecture, which is responsible for integrating the feature information processed by the cross-scale channel attention collaborative module CS-CAC, thereby effectively unifying the features extracted at different scales and constructing a complete view of global and detailed information. Comprehensive feature fusion not only enhances the model's understanding of the input data, but also improves the model's adaptability and performance when processing various scales and complex scenes. Then, a cross-scale adaptive feature fusion module CSAF was designed. Based on the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, this module cross-scales the features in the spatial dimension, further enhancing the model's ability to integrate multi-scale information. Finally, Mask RCNN was selected as the decoder in the Decoder stage. By fusing the feature maps of different stages, the final detection feature map was obtained to complete the accurate detection and identification of FPC defects. Through these innovative modules and strategies, a Transformer-based flexible circuit board defect detection method can better cope with the challenges of multi-scale information interaction and improve the performance and robustness in FPC defect detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 This is an overall flow chart of a flexible circuit board defect detection method based on Transformer.
[0066] Figure 2 Flowchart of the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA;
[0067] Figure 3 Flowchart of the cross-scale channel attention coordination module CS-CAC;
[0068] Figure 4 Flowchart of the cross-scale adaptive feature fusion module CSAF. DETAILED DESCRIPTION
[0069] In order to simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Of course, they are only examples and are not intended to limit the present invention.
[0070] In the embodiment, a flexible circuit board defect detection method based on Transformer is proposed, which belongs to a cross-scale collaborative Transformer defect detection method (Cross Scale Collaborative Transformer, CSCFormer). The key to this method lies in the extraction and fusion of multi-scale features. First, the multi-head self-attention mechanism is improved, and a cross-scale collaborative multi-head self-attention mechanism (Cross-Scale Collaborative Multi-Head Self-Attention, CSC-MHSA) is designed to realize the collaborative interaction of multi-scale information. Then, on the basis of the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, a cross-scale channel attention collaborative module (Cross-Scale Channel Attention Collaborative, CS-CAC) and a cross-scale adaptive feature fusion module (Cross-Scale Aadaptive Feature Fusion, CSAF) are designed to realize the collaborative interaction of cross-scale information in channel and spatial dimensions. The cross-scale channel attention collaborative module CS-CAC increases the multi-scale information of each channel feature, so that the model can more accurately perform cross-scale channel attention collaborative interaction, thereby improving the perception of features of different scales. The cross-scale adaptive feature fusion module CSAF realizes cross-scale adaptive feature information interaction in the spatial dimension, fuses global and local features, and thus improves the model's defect recognition and anti-interference capabilities. Finally, in order to reduce computational complexity and enhance real-time performance, depthwise separable convolution is used as the convolution operation in the module.
[0071] A Transformer-based flexible circuit board defect detection method proposed in this invention was tested on the NEU-DET public dataset. The test results showed that the AP50 (average accuracy at IoU=0.5) was improved by 3.7% compared with the baseline model. Among them, the average accuracy of small, medium and large targets was improved by 3.6%, 1.9% and 3.0% respectively, and the average recall was improved by 2.6%, 0.7% and 6.5% respectively. On the self-built FPCSD2023 dataset, a Transformer-based flexible circuit board defect detection method improved the AP50 by 6.1% compared with the baseline model, and the average accuracy of small, medium and large targets was improved by 5.4%, 3.0% and 4.5% respectively, and the average recall was improved by 7.4%, 4.9% and 6.3% respectively. In addition, it showed good real-time performance at 23FPS. Finally, an FPC defect intelligent detection system was established, and a series of physical experiments were successfully carried out.
[0072] The overall process of a flexible circuit board defect detection method based on Transformer is shown in the attached figure. Figure 1 As shown. The standard Encoder-Decoder structure is adopted, and it is inspired by ACNet (Ding et al., 2019) to realize the interaction of information at different scales of the image. When constructing the encoder stage, Swin Transformer was selected as the backbone network based on its outstanding performance in image feature extraction tasks. Although Swin Transformer performs well in extracting local and global features, due to the complexity of multi-scale features in the image, the single feature extraction at each stage cannot fully meet the needs of multi-scale information, especially in the FPC defect detection task. Since FPC defects have specific shape or texture features at different scales, the full capture of multi-scale information is particularly critical. The lack of multi-scale information may lead to insufficient adaptability of the algorithm at different observation distances or viewing angles, resulting in problems such as false detection and missed detection. In order to address this challenge, a series of innovative modules are introduced. The process of a Transformer-based flexible circuit board defect detection method includes the following steps:
[0073] Step S1: Import the FPC image data into the encoder for PFC defect recognition, and the encoder encodes the pixel information of the image, extracts the image features, and outputs the FPC image feature data; the FPC image feature data includes the channel dimension information collaborative interaction features of scale 1, scale 2 and scale 3, which are respectively , and ;
[0074] in, is the cooperative interaction feature of channel dimension information at scale one;
[0075] It is the collaborative interaction feature of channel dimension information of scale 2;
[0076] It is the collaborative interaction feature of channel dimension information of scale three;
[0077] When constructing the encoder, Swin Transformer is selected as the backbone network, and FPC defect recognition is performed through four stages;
[0078] Step S2: The , and , and the fourth stage feature Scale_4 obtained by the encoder when performing PFC defect recognition is imported into the feature pyramid network FPN, and the feature pyramid network FPN is integrated to unify the features extracted at different scales and output the features of five stages, respectively. , , , and ;
[0079] in, The first stage features output by the feature pyramid network FPN;
[0080] The second stage features output by the feature pyramid network FPN;
[0081] The third stage features output by the feature pyramid network FPN;
[0082] The fourth stage features output by the feature pyramid network FPN;
[0083] The fifth stage feature output by the feature pyramid network FPN;
[0084] Step S3: The , , , and The cross-scale adaptive feature fusion module CSAF is introduced. The cross-scale adaptive feature fusion module CSAF performs cross-scale fusion of features in the spatial dimension based on the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, and outputs the cross-scale fused FPC image data F csaf ;
[0085] Step S4: The cross-scale fused FPC image data F obtained in step S3 is csaf The decoder is imported and fused by the decoder to obtain the final FPC image detection feature map, thereby completing the detection and identification of FPC defects.
[0086] In step S1, the encoder performs image feature extraction, which specifically includes the following steps:
[0087] Step S11: importing the FPC image data into the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA fuses the key features of the FPC image at scales 1, 2, and 3 to obtain the scale 1, scale 2, and scale 3 information collaborative interaction features, which are respectively , and ; Among them, ici represents information collaborative interaction;
[0088] Step S12: The , and The cross-scale channel attention collaborative module CS-CAC is introduced. The cross-scale channel attention collaborative module CS-CAC combines the channel attention mechanism and cross-scale information transfer to obtain the channel dimension information collaborative interaction features of scale one, scale two and scale three, respectively. , and .
[0089] The cross-scale collaborative multi-head self-attention mechanism CSC-MHSA in step S11 is as follows:
[0090] In order to ensure the optimal collaborative interaction of cross-scale information, a cross-scale collaborative multi-head self-attention mechanism CSC-MHSA is designed, which is different from the original multi-head attention mechanism MHSA in calculation method. This mechanism effectively integrates the key features of scales 1, 2, and 3 by fusing the query vector (Q). This design allows the module to capture and integrate important visual information at multiple scales, improving the system's ability to understand complex scenes and accurately detect defective targets. The process of the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA is shown in the attached figure. Figure 2 As shown, the cross-scale adaptive feature fusion module CSAF in step S11 performs the following steps when performing cross-scale fusion, specifically:
[0091] Step S111: import the FPC image data and extract scale 1, scale 2 and scale 3 features respectively, , and ;
[0092] in, Represents the height, width and number of color channels of scale-one feature in three-dimensional space;
[0093] Represents the height, width, and number of color channels of scale 2 features in three-dimensional space;
[0094] Represents the height, width and number of color channels of the three scale features in three-dimensional space;
[0095] Step S112: The , and The cross-scale channel attention collaborative module CS-CAC based on the cross-scale collaborative multi-head self-attention mechanism is respectively imported for sequence conversion, and then the projection matrices of scales 1, 2 and 3 are respectively used. , and , calculate the Q, K and V vectors of scale one, scale two and scale three respectively, , and , and output;
[0096] in, represents a three-dimensional matrix,
[0097] Q represents the query vector, which is used to determine the focus in the attention calculation;
[0098] By interacting with K, it decides what information to extract from V;
[0099] K is the key vector associated with the input, which is compared with Q to determine the importance of different parts of the input;
[0100] V is the actual value vector, which contains the specific information of the input. According to the interaction results of Q and K, it is determined which information in V should be paid attention to and extracted;
[0101] is the query matrix of scale one, which is used to project the input data into the query space of scale one to generate the query vector;
[0102] is the Key matrix of scale one, which is used to project the input data of scale one into the key space and generate the key vector;
[0103] It is the Value matrix of scale one, which is used to project the input data of scale one into the value space and generate the value vector;
[0104] is the query matrix of scale 2, which is used to project the input data of scale 2 into the query space and generate the query vector;
[0105] is the Key matrix of scale 2, which is used to project the input data of scale 2 into the key space and generate the key vector;
[0106] It is the Value matrix of scale 2, which is used to project the input data of scale 2 into the value space and generate a value vector;
[0107] is the query matrix of scale 3, which is used to project the input data of scale 3 into the query space and generate the query vector;
[0108] is the Key matrix of scale three, which is used to project the input data of scale three into the key space and generate the key vector;
[0109] It is the Value matrix of scale three, which is used to project the input data into the value space of scale three and generate the value vector;
[0110] Step S113: Use the concatenate operation to connect the Q vectors of scale 1, scale 2, and scale 3 outputted from step S112, and use a 1×1 convolution to perform channel adjustment to obtain the fused vector Q Fusion ; Using the fusion vector Q Fusion Perform matrix multiplication with the transpose of the K vector of scale 1, scale 2 and scale 3 outputted in step S112, and calculate the self-attention matrices of scale 1, scale 2 and scale 3, which are respectively , and , and output;
[0111] Step S114: The self-attention matrices of scale 1, scale 2, and scale 3 obtained in step S113 are , and Multiply with the V vector of each scale to obtain the collaborative features; use a fully connected layer to adjust the dimension and map the output features of scales 1, 2, and 3 to Get in space , and . Where ici stands for information collaborative interaction (ici).
[0112] The formula for the above steps is as follows: ; ; ; ; ; ; ; ; ; ;
[0113] Where T represents the transpose of the matrix, d head Represents the dimension of each head, Concat represents the concatenate operation, Conv represents 1×1 convolution, and FC represents the fully connected layer.
[0114] The cross-scale channel attention coordination module CS-CAC in step S12 is as follows:
[0115] In the field of images, the features that can be captured by different channels have their own unique information content, so these features can be weighted and screened to improve the model's feature extraction ability. SENet introduces a channel attention mechanism to achieve weighted processing of each channel's features, thereby effectively improving the efficiency of feature extraction. The core idea of this mechanism is to adaptively adjust the weight of each channel according to its importance, so that the model pays more attention to the features that are decisive for classification or detection tasks. SENet's channel attention mechanism has become an important technology for image processing. It first calculates the importance of each channel through global information, and then performs weighted operations on the feature maps of each channel according to these importance weights, highlighting important features and suppressing irrelevant information. This mechanism enables the model to adapt to different input images more flexibly and to extract and utilize key information in the image more effectively.
[0116] The channel attention mechanism of SENet dynamically adjusts the importance of each channel by calculating the attention weight of each channel, so that the network can more effectively capture the correlation and importance between image features. However, CAM adopts the method of global pooling layer and fully connected layer. Although it can extract the overall image features, it leads to the loss of multi-scale features in the image. In actual image processing tasks, multi-scale features are crucial for accurately distinguishing objects of different scales. In order to solve the above problems, a new channel attention calculation method is proposed, namely the cross-scale channel attention collaborative module CS-CAC. This module is mainly divided into two parts: the image-to-tokens scale mapping part (Image-to-Tokens ScaleMapping, ITSM) and the channel attention collaborative part (Channel Attention Collaborative, CAC). The core idea of the cross-scale channel attention collaborative module CS-CAC is to combine the channel attention mechanism and cross-scale information transfer, so that the model can better understand and utilize the feature information at different scales, thereby enhancing the multi-scale representation ability of the target. The detailed process of the cross-scale channel attention collaborative module CS-CAC described in step S12 is as shown in the attached figure. Figure 3The cross-scale channel attention collaborative module CS-CAC in step S12 executes steps S121 to S125, wherein steps S121 to S123 are Image-to-Tokens Scale Mapping (ITSM), and steps S124 to S125 are Channel Attention Collaborative (CAC).
[0117] (1) Image-to-Tokens Scale Mapping (ITSM)
[0118] The ITSM part aims to better capture the multi-scale information in the image by converting a single image into feature representations of three different scales through processing. This processing is crucial for defect detection tasks because defects in images may have different scales and sizes, requiring the model to accurately identify and locate at different scales. Therefore, the design of ITSM provides multi-scale feature representations to enhance the model's ability to understand images. In addition, the ITSM part is also responsible for converting the features of each channel into a one-dimensional token form to meet the input requirements of the Transformer model. Through this conversion, the image changes from H×W×C to C×C1, where C represents the number of channels of the feature and C1 represents the dimension size of each token. This form of input not only conforms to the structure of the Transformer model, but also helps to improve the processing efficiency and generalization ability of the model, so as to better cope with various image processing tasks.
[0119] Step S121: The step S11 obtained , and Perform in-depth pooling processing and output pooling results of scale 1, scale 2, and scale 3;
[0120] Step S122: The pooling results of scale 1, scale 2 and scale 3 obtained in step S121 are merged in the second dimension to form multi-scale pooling features of scale 1, scale 2 and scale 3, which are respectively , and , and output; where the subscript mspf represents that the feature belongs to the multi-scale pooling feature;
[0121] Step S123: Use a fully connected layer to transform the , and They are mapped to a dimension C1 higher than 2L, and three serialized features are obtained, which are , and , and output;
[0122] The step S121 is , and The specific process of deep pooling is as follows: four pooling scales are set, namely M1×N1, M2×N2, M3×N3, and M4×N4;
[0123] Then the pooled features Flattened into a one-dimensional feature vector , where M and N represent the sizes of different pooling factors; the flattened vectors are merged in the second dimension to form pooling results of scale one, scale two, and scale three, respectively. , , , , and , where L=M1N1+M2N2+M3N3+M4N4, and the subscripts mp and ap represent maximum pooling and average pooling.
[0124] This series of steps aims to extract multi-scale information from image features and transform the features of each scale in a way that meets the input requirements of the Transformer model, so as to more effectively utilize this multi-scale information for subsequent processing and analysis.
[0125] (2) Channel Attention Collaborative (CAC)
[0126] Through the ITSM part, each channel of the feature map is serialized to ensure that the features are consistent with the input features of the Transformer model. Next, the features of all channels are globally modeled using the standard stacked Transformer structure, which helps the model better understand the global information in the image. After this step, a multi-layer perceptron (MLP) and a sigmoid activation function are used to calculate the channel attention vector of each scale, thereby further enhancing the model's attention to features of different scales. To ensure that the model can accurately distinguish features from different scales and avoid inaccurate information processing, a learnable scale type encoding (STE) is introduced. This encoding is similar to the position encoding in the Transformer.
[0127] Step S124: the value obtained in step S123 , and Add scale type encoding; input the features of scales 1, 2, and 3 into N Transformer modules, and use the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA to realize cross-scale collaborative interaction of channel information. The features obtained are , and , and output; where cici represents channel information collaborative interaction; (channel information collaborative interaction, cici); these features are processed by multiple layers of Transformer and are represented more richly and accurately; and output , and ;
[0128] The calculation process of each Transformer module is expressed as follows: ; ;
[0129] Among them, z l Represents the input of the lth module, the value range of l is 1 to N, LN represents the normalization layer, CSCMHSA represents the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, and MLP represents the multi-layer perceptron.
[0130] Step S125: Use a multilayer perceptron and a Sigmoid activation function to calculate the value obtained in step S124 , and , we get the channel attention vectors of scale one, scale two, and scale three, respectively , and ; Then, the original features , and Multiply the channel attention vector by channel to obtain the channel dimension information collaborative interaction features of scale one, scale two, and scale three, respectively. , and , and output.
[0131] This strengthens the importance of each channel in the feature map and highlights the key information. It helps to focus on the most important channels in each scale feature map, thereby enhancing the model's ability to express and distinguish key features.
[0132] The above steps are expressed as follows: ; ; ; ; ; ; ;
[0133] in, Indicates the multiplication of corresponding channels, It indicates the addition of corresponding elements, and STE indicates the scale type encoding.
[0134] Since the cross-scale channel attention collaboration module CS-CAC performs self-attention calculations in the channel dimension, the additional computation required is minimal. This design enables the cross-scale channel attention collaboration module CS-CAC to maintain a high level of computational efficiency while enhancing model performance. Even in resource-constrained situations, the introduction of the cross-scale channel attention collaboration module CS-CAC will not have a significant impact on the overall efficiency of the model. This is especially important for scenarios where target detection tasks need to be run in an environment with limited computing resources. The lightweight design of the cross-scale channel attention collaboration module CS-CAC ensures that the model can fully utilize the multi-scale features in the image and improve detection performance while maintaining high efficiency.
[0135] Adaptively Spatial Feature Fusion (ASFF)
[0136] Before introducing the cross-scale adaptive feature fusion module CSAF, we first introduce the adaptive spatial feature fusion ASFF, because the cross-scale adaptive feature fusion module CSAF uses a variant of the adaptive spatial feature fusion ASFF. Referring to the ASFF part proposed by Liu et al. (2019), first, after the coordinated interaction of the channel dimension information of scales 1, 2, and 3, it is necessary to and Features and Features are aligned to fuse scales 1, 2, and 3. and The features are adjusted to the same size through upsampling (resize) operation. Then, three parameters α, β, and γ are introduced. These three parameters are learned to represent the importance of each scale feature, and the learned parameters are normalized using the Sigmoid activation function. Finally, after multiplying the three adjusted features with the corresponding parameter values, the features of scales 1, 2, and 3 are weighted and summed bit by bit to obtain the cross-scale channel attention coordination module CS-CAC. This process helps to effectively integrate information of features at different scales.
[0137] The above steps are expressed as follows: ;
[0138] in, Indicates resizing from Scale2 features to Scale1 features. Indicates resizing from Scale3 feature to Scale1 feature size. mn , β mn , γ mn represents the values of α, β, γ at position (m, n), satisfying α mn +β mn +γ mn =1 and , Represents the Sigmoid activation function.
[0139] The cross-scale adaptive feature fusion module CSAF in step S3 is as follows:
[0140] Some current feature fusion methods use different attention mechanisms to achieve multi-scale feature fusion. Some use channel attention mechanisms, while others combine channel attention mechanisms and spatial attention mechanisms. However, these methods have some limitations: the channel attention mechanism may lead to the loss of multi-scale features of the image because it mainly focuses on the channel dimension and ignores the information of the spatial dimension; while the convolution-based spatial attention mechanism can capture local information, but cannot obtain the global receptive field and may ignore the global context in the image. In addition, some methods only perform feature fusion at the last downsampling, which means that only global features are considered and local features are ignored, which may limit the model's ability to understand and express image details. Moreover, currently only using feature pyramid networks FPN for feature fusion may cause information loss. Although features at different levels are fused through top-down and bottom-up feature propagation, some detailed information will be lost during the feature propagation process. In addition, feature pyramid networks FPNs have difficulty in handling situations with large scale differences, such as when there are very small and very large targets at the same time, feature pyramid networks FPNs may face difficulties.
[0141] Therefore, it is very necessary to design a cross-scale adaptive feature fusion module CSAF. The design idea of this module is that after the features of each stage pass through the cross-scale channel attention coordination module CS-CAC, the feature pyramid network FPN is used to fuse the features of the four stages, and then the features of the two adjacent stages (different scales) are sent to the cross-scale adaptive feature fusion module CSAF for feature fusion. In the cross-scale adaptive feature fusion module CSAF, the weights between the features of the two stages are adaptively adjusted, and then the scales of the two adjacent stages are collaboratively interacted with global information in the spatial dimension, and then the cross-scale features are fused. This processing method can help the model better fuse feature information from different levels and scales. The cross-scale adaptive feature fusion module CSAF refers to ASFF (Liu et al., 2019) to design an ASFF-S that can effectively and adaptively fuse these features across scales. An adaptive weight mechanism is introduced to adaptively adjust the weights between the features of the two stages, so that the model can dynamically learn the importance of each scale feature, thereby better fusing information of different scales. This fusion module of CSAF helps the model better understand the semantic information in the image. The overall structure of the cross-scale adaptive feature fusion module CSAF is shown in the attached figure. Figure 4 In step S3, the cross-scale adaptive feature fusion module CSAF performs the following steps when performing cross-scale fusion, specifically:
[0142] Step S31: for the data obtained in step S2 , , , and , the adjacent i-th scale and j-th scale features and The feature uses 1×1 convolution to adjust the number of channels, and adjusts the features of another layer to the same size as the target feature by upsampling or downsampling. Then, two learnable adaptive weights are introduced in the feature fusion module ASFF-S. and , and then processed by the Sigmoid function, it is multiplied with the features of the two scales after resizing to obtain two adaptively adjusted features, which are and , and output;
[0143] The above steps are expressed as follows: ; ;
[0144] in, Represents the feature resize from the jth stage to the feature size of the ith stage. , is a learnable parameter. , express , The value at position (m, n). , satisfy and . Represents the Sigmoid activation function.
[0145] Step S32: for the step S31 obtained and The cross-scale adaptive feature fusion module CSAF uses deep separable convolution to calculate self-attention in the spatial dimension, and introduces implicit position coding to implicitly encode the position information. Then, the feature information after implicit position coding is residually connected with the original feature to obtain two features, namely and , and output; where pse represents position and scale encoding;
[0146] The computational complexity and parameter size of depth-wise separable convolution are expressed as follows: ; ;
[0147] The amount of computation and parameters of standard convolution is expressed as follows: ; ;
[0148] The parameter ratio and computation ratio of depthwise separable convolution to standard convolution are expressed as follows: ; ;
[0149] The above steps are expressed as follows: ; ;
[0150] in, right Perform depth-wise separable convolution operations; Express Perform depth-wise separable convolution operation. , Respectively represent characteristics , STE encoding.
[0151] Step S33: for the step S32 obtained and In the spatial dimension, cross-scale information collaborative interaction is carried out, and a spatial reduction module is introduced. A normalization layer is used to obtain two features after spatial information collaborative interaction, which are and ;
[0152] Step S34: Use a 1×1 convolutional layer to adjust the obtained in step S33 and The dimension of the channel is fused into a single feature; at the same time, the local feature F of the original feature is obtained through the depth-separable convolution local , and the global feature F global Perform residual connection fusion; finally, perform normalization through the Batch Normalization layer to obtain the cross-scale fused FPC image data F csaf ;
[0153] The above steps are expressed as follows: ; ; ; ;
[0154] Among them, F global and F local Represents global features and local features respectively, and CDRC represents a series of operation steps (Conv 1×1 , DWC 3×3 , Relu and Conv 1×1 ), BN represents Batch Normalization operation.
Claims
1. A flexible circuit board defect detection method based on Transformer, characterized in that: The following steps are involved: Step S1: Import the FPC image data into the encoder for PFC defect recognition, and the encoder encodes the pixel information of the image, extracts the image features, and outputs the FPC image feature data; the FPC image feature data includes the channel dimension information collaborative interaction features of scale 1, scale 2 and scale 3, which are respectively and in, is the cooperative interaction feature of channel dimension information at scale one; It is the collaborative interaction feature of channel dimension information of scale 2; It is the collaborative interaction feature of channel dimension information of scale three; When constructing the encoder, Swin Transformer is selected as the backbone network, and FPC defect recognition is performed through four stages; Step S2: The and The fourth stage feature Scale_4 obtained by the encoder when performing PFC defect recognition is imported into the feature pyramid network FPN, and the feature pyramid network FPN integrates the features extracted at different scales and outputs the features of five stages, namely Scale_1 cfp ,Scale_2 cfp ,Scale_3 cfp ,Scale_4 cfp and Scale_5 cfp ; Among them, Scale_1 cf The first stage features output by the feature pyramid network FPN; Scale_2 cfp The second stage features output by the feature pyramid network FPN; Scale_3 cfp The third stage features output by the feature pyramid network FPN; Scale_4 cfp The fourth stage features output by the feature pyramid network FPN; Scale_5 cfp The fifth stage feature output by the feature pyramid network FPN; Step S3: Scale_1 obtained in step S2 cfp ,Scale_2 cfp ,Scale_3 cfp ,Scale_4 cfp and Scale_5 cfp The cross-scale adaptive feature fusion module CSAF is introduced. The cross-scale adaptive feature fusion module CSAF performs cross-scale fusion of features in the spatial dimension based on the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, and outputs the cross-scale fused FPC image data F csaf ; Step S4: The cross-scale fused FPC image data F obtained in step S3 is csaf Importing the decoder, and fusing it with the decoder to obtain the final FPC image detection feature map, and completing the detection and identification of FPC defects; In step S1, the encoder performs image feature extraction, which specifically includes the following steps: Step S11: importing the FPC image data into the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA, the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA fuses the key features of the FPC image at scales 1, 2, and 3 to obtain the scale 1, scale 2, and scale 3 information collaborative interaction features, which are Scale1, Scale2, and Scale3, respectively. ici Scale2 ici and Scale3 ici ; Among them, ici represents information collaborative interaction; Step S12: Scale1 obtained in step S11 ici Scale2 ici and Scale3 ici The cross-scale channel attention collaborative module CS-CAC is introduced. The cross-scale channel attention collaborative module CS-CAC combines the channel attention mechanism and cross-scale information transfer to obtain the channel dimension information collaborative interaction features of scale one, scale two and scale three, respectively. and The cross-scale adaptive feature fusion module CSAF performs the following steps when performing cross-scale fusion, specifically: Step S31: Scale_1 obtained in step S2 cfp ,Scale_2 cfp ,Scale_3 cfp ,Scale_4 cfp and Scale_5 cfp , the adjacent i-th scale and j-th scale features and The feature uses 1×1 convolution to adjust the number of channels, and adjusts the other layer of features to the same size as the target feature through upsampling or downsampling. Then, two learnable adaptive weights δ and ε are introduced in the feature fusion module ASFF-S. After being processed by the Sigmoid function, they are multiplied with the resized features of the two scales to obtain two adaptively adjusted features, which are and And output; Step S32: for the step S31 obtained and The cross-scale adaptive feature fusion module CSAF uses deep separable convolution to calculate self-attention in the spatial dimension, and introduces implicit position coding to implicitly encode the position information. Then, the feature information after implicit position coding is residually connected with the original feature to obtain two features, namely and And output; where pse represents position and scale encoding; Step S33: for the step S32 obtained and In the spatial dimension, cross-scale information collaborative interaction is carried out, and a spatial reduction module is introduced. A normalization layer is used to obtain two features after spatial information collaborative interaction, which are and Step S34: Use a 1×1 convolutional layer to adjust the obtained in step S33 and The dimension of the channel is converted into a single feature. At the same time, the local feature F of the original feature is obtained through the depth-separable convolution. local , and the global feature F global Perform residual connection fusion; finally, perform normalization through the BatchNormalization layer to obtain the cross-scale fused FPC image data F csaf .
2. According to the Transformer-based flexible circuit board defect detection method of claim 1, it is characterized in that: The decoder Decoder selects Mask RCNN as the decoder.
3. According to claim 2, a flexible circuit board defect detection method based on Transformer is characterized in that: The cross-scale collaborative multi-head self-attention mechanism CSC-MHSA performs the following steps when fusing the key features of the FPC image at scales 1, 2, and 3: Step S111: import the FPC image data and extract scale 1, scale 2 and scale 3 features respectively, and in, Represents the height, width and number of color channels of scale-one feature in three-dimensional space; Represents the height, width, and number of color channels of scale 2 features in three-dimensional space; Represents the height, width and number of color channels of the three scale features in three-dimensional space; Step S112: The and The cross-scale channel attention collaborative module CS-CAC based on the cross-scale collaborative multi-head self-attention mechanism is respectively imported for sequence conversion, and then the projection matrices of scales 1, 2 and 3 are respectively used. and Calculate the Q, K and V vectors of scale 1, scale 2 and scale 3 respectively, which are (Q scal , K scal , V scale1 )、(Q scal , K scale2 , V scal ) and (Q scale3 , K scale3 , V scale3 ), and output; in, represents a three-dimensional matrix, Q represents the query vector, which is used to determine the focus in the attention calculation; By interacting with K, decide what information to extract from V; K is the key vector associated with the input, which is compared with Q to determine the importance of different parts of the input; V is the actual value vector, which contains the specific information of the input. According to the interaction results of Q and K, it is determined which information in V should be paid attention to and extracted; It is the query matrix of scale one, which is used to project the input data into the query space of scale one to generate the query vector; is the Key matrix of scale one, which is used to project the input data of scale one into the key space and generate the key vector; It is the Value matrix of scale one, which is used to project the input data of scale one into the value space and generate a value vector; It is the query matrix of scale 2, which is used to project the input data of scale 2 into the query space and generate the query vector; It is the Key matrix of scale 2, which is used to project the input data of scale 2 into the key space and generate the key vector; It is the Value matrix of scale 2, which is used to project the input data of scale 2 into the value space and generate a value vector; It is the query matrix of scale 3, which is used to project the input data of scale 3 into the query space and generate the query vector; It is the Key matrix of scale three, which is used to project the input data of scale three into the key space and generate the key vector; It is the Value matrix of scale three, which is used to project the input data into the value space of scale three and generate the value vector; Step S113: Use the concatenate operation to connect the Q vectors of scale 1, scale 2, and scale 3 outputted from step S112, and use a 1×1 convolution to perform channel adjustment to obtain the fused vector Q Fusion ; Using the fusion vector Q Fusion Perform matrix multiplication with the transpose of the K vector of scale 1, scale 2, and scale 3 outputted in step S112 to calculate the self-attention matrices of scale 1, scale 2, and scale 3, which are Attention scale1 、Attention scale2 and Attention scale3 , and output; Step S114: The self-attention matrices of scale 1, scale 2, and scale 3 obtained in step S113 are scale1 、Attention scale2 and Attention scale3 Multiply with the V vector of each scale to obtain the collaborative features; use a fully connected layer to adjust the dimension and map the output features of scales 1, 2, and 3 to Get Scale1 in space ici Scale2 ici and Scale3 ici .
4. According to claim 3, a flexible circuit board defect detection method based on Transformer is characterized in that: The cross-scale channel attention coordination module CS-CAC specifically performs steps S121 to S125 when performing cross-scale information coordination and complementation: Step S121: Scale1 obtained in step S11 ici Scale2 ici and Scale3 ici Perform in-depth pooling processing and output pooling results of scale 1, scale 2, and scale 3; Step S122: The pooling results of scale 1, scale 2 and scale 3 obtained in step S121 are merged in the second dimension to form multi-scale pooling features of scale 1, scale 2 and scale 3, which are respectively and And output; where the subscript mspf represents that the feature belongs to the multi-scale pooling feature; Step S123: Use a fully connected layer to transform the and They are mapped to a dimension C1 higher than 2L, and three serialized features are obtained, which are and And output; Step S124: Scale1 obtained in step S123 tokenized , Scale2 tokenized and Scale3 tokenized Add scale type encoding; input the features of scales 1, 2, and 3 into N Transformer modules, and use the cross-scale collaborative multi-head self-attention mechanism CSC-MHSA to realize cross-scale collaborative interaction of channel information. The features obtained are and And output; where cici represents the collaborative interaction of channel information; Step S125: Use a multilayer perceptron and a Sigmoid activation function to calculate the value obtained in step S124 and The channel attention vectors of scale 1, scale 2, and scale 3 are obtained, respectively and Then, the original feature Scale1 in Scale2 in and Scale3 in Multiply the channel attention vector by channel to obtain the channel dimension information collaborative interaction features of scale one, scale two, and scale three, respectively. and And output.
5. The flexible circuit board defect detection method based on Transformer according to claim 4 is characterized in that: The step S121 is to ici Scale2 ici and Scale3 ici In-depth pooling processing is performed by applying multi-scale average pooling and multi-scale maximum pooling techniques to adapt to the features of these different scales.
6. The flexible circuit board defect detection method based on Transformer according to claim 5 is characterized in that: The step S121 is to ici Scale2 ici and Scale3 ici The specific process of deep pooling is as follows: Set four pooling scales: M1×N1, M2×N2, M3×N3, and M4×N4; Then the pooled features Flattened into a one-dimensional feature vector Where M and N represent the sizes of different pooling factors; The flattened vectors are merged in the second dimension to form pooling results of scale one, scale two, and scale three, respectively. and Where L = M1N1 + M2N2 + M3N3 + M4N4, and the subscripts mp and ap are used to represent maximum pooling and average pooling.
Citation Information
Patent Citations
Segmentation algorithm for detecting laser welding defect of power battery safety valve
CN112184686A
Product surface defect detection method in industrial scene
CN114359245A
Aerial photography target detection method based on composite backbone network and multiple prediction heads
CN115035429A
Steel surface defect detection method and system
CN117523298A
X-ray weld defect intelligent evaluation method based on model self-optimization
CN117557534A
