A defect detection method based on joint optimization and mixed attention feature fusion

Abstract

This invention relates to a segmentation-classification-based defect detection method. Addressing the insufficient feature extraction capability of two-stage defect detection algorithms, this invention proposes a hybrid attention feature fusion module integrated into an encoder-decoder segmentation network. This allows the model to better utilize global contextual information and reconstruct pixel-level segmentation maps using extracted deep features. Furthermore, this invention proposes a multi-receptive-field spatial attention module, leveraging dilated convolution to enhance the receptive field and extract spatial attention weights, effectively improving the model's ability to extract minute features. To address the low training efficiency of two-stage defect detection models, this invention proposes a joint optimization framework, utilizing a constructed joint loss function for end-to-end model training. Experiments show that the improvements proposed in this invention effectively improve the detection accuracy of defect tasks.

Description

Technical Field

[0001] The technical solution of this invention relates to the fields of deep learning, convolutional neural networks and defect detection, specifically a defect detection method based on joint optimization and hybrid attention feature fusion. Background Technology

[0002] Defect detection technology is an indispensable and important means in the quality control process of modern industrial production. Traditional machine vision inspection methods mainly utilize the different properties of defects on the workpiece surface to formulate reasonable imaging schemes, and process manually set features through machine learning-based image processing algorithms to extract defect information that may be contained on the workpiece surface. Such methods have been widely researched and applied in the field of industrial production.

[0003] With the deepening research in the field of deep learning, deep neural network models, represented by Convolutional Neural Networks (CNNs), have been widely applied in the field of defect detection, demonstrating outstanding performance in defect feature extraction and defect classification. The defect detection problem can be simply summarized as a binary classification problem of identifying whether an image contains a defect. Based on the classification problem and the need for information such as defect shape, type, and location, research areas such as defect localization and defect segmentation can be derived.

[0004] Existing classification-based convolutional neural network surface defect detection methods can be broadly summarized into "one-stage" methods that use the original image for classification and "two-stage" methods that combine segmentation and localization tasks. These two types of methods mainly suffer from the following shortcomings:

[0005] (1) The "single-segment" method that uses the original image for classification often has a relatively simple and shallow network structure. The feature extraction capability of such network models is slightly insufficient when facing defect types with complex shapes, resulting in unsatisfactory classification accuracy. In addition, the defect area in actual industrial scenarios is often small relative to the acquired image, and such methods cannot effectively utilize spatial and positional information, easily ignoring small defects. Adding an attention mechanism and feature fusion method can effectively improve the feature extraction capability of convolutional neural networks while making the network more focused on the defect part. Therefore, this invention adds a feature fusion method based on an attention mechanism to the convolutional neural network.

[0006] (2) Two-stage classification models typically employ a non-end-to-end training mode, where the segmentation branch is trained first and its network parameters are saved. Then, the model weights with better segmentation results are selected, loaded into the network, and the classification branch is trained. While this training mode ensures good training for both stages, it also suffers from low training time efficiency and high computational resource consumption. The joint optimization method proposed in this invention enables end-to-end training of the model and improves its classification accuracy. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to propose a defect detection method based on joint optimization and hybrid attention feature fusion. This method provides a spatial attention mechanism based on multiple receptive fields, effectively enhancing the model's ability to extract features from minute defects. Furthermore, it combines a channel attention mechanism to construct a hybrid attention feature fusion module, replacing the feature fusion method of long-jump connections at different scales in traditional autoencoder models, thus better utilizing global contextual information. This method employs a joint optimization framework for end-to-end training, which significantly improves the accuracy of defect detection tasks.

[0008] The technical solution adopted by this invention to solve this technical problem is: to provide a defect detection method based on joint optimization and hybrid attention feature fusion, the steps of which are as follows:

[0009] Step 1: Acquire images of the surface of the workpiece to be tested, preprocess the acquired images, set the real labels used for training, and build a network model, which consists of two parts: a segmentation network and a classification network.

[0010] The second step is to input the preprocessed image into the model for training, set the optimization parameters and the number of iterations, and the model output is a pixel-level segmentation map of the defect information and the corresponding defect type.

[0011] Step 3: Save the trained model weights and use the model to detect surface defects on the workpiece.

[0012] In the first step described above, the acquired image is downsampled to a size of 512×512 pixels, and the original RGB three-channel image is converted to a single-channel image, i.e., converted to grayscale. After standardization, it is input into the model. The ground truth labels required for training the segmentation network are pixel-level labels, i.e., pixels in the corresponding defect area are labeled as positive samples, and pixels in the normal area are labeled as negative samples. In addition, corresponding category indicator labels are assigned according to the type of surface defect of the workpiece to train the classification network.

[0013] The segmentation network consists of an encoder-decoder backbone and a hybrid attention feature fusion module. The encoder part includes four consecutive downsampling operations with a stride of 2. The feature maps extracted from each layer are input to the hybrid attention feature fusion module, where they are concatenated with the feature maps of the same level and resolution reconstructed by the decoder structure before participating in subsequent convolution calculations. Finally, the decoder outputs a pixel-level segmentation map with the same size as the input image, indicating the location and shape of defects.

[0014] The input to the classification network is a 32×32×256 feature map output by the segmentation network. The feature is enhanced by the multi-receptive field spatial attention module and transformed into a one-dimensional feature vector through convolution and pooling operations with 5×5 convolution kernels. The fully connected layer then performs classification operations.

[0015] Specifically, before each downsampling step, the obtained feature map is concatenated with the low-dimensional feature map decoded by the corresponding network layer of the decoder using long-skip connections. This concatenation is achieved through a hybrid attention feature fusion module, preserving complete semantic information while restoring the image to its original resolution. After Global Average Pooling (GAP) and Global Max Pooling (GMP), the concatenation is performed with the one-dimensional feature vector output by the classification network to guide the classification network's results. The resulting 66×1 feature vector is then input into a fully connected layer and subjected to Softmax operations to obtain the confidence score for each class.

[0016] The hybrid attention feature fusion module consists of a multi-receptive-field spatial attention module and a channel attention module. In the multi-receptive-field spatial attention module, the input feature map X of dimension H×W×C is activated by the non-linear activation function ReLU after convolution operations under different receptive fields. It is then concatenated into an H×W×3 feature map, which is then compressed to 1 channel by a 1×1 convolution. After activation by the Sigmoid function, it is multiplied with the original input feature map to obtain the attention-weighted feature map X'.

[0017]

[0018] Where concat[·;·;·] represents the concatenation operation. ReLU(·) represents the ReLU activation function, which is based on a 3×3 kernel and performs convolution operations with different receptive fields.

[0019] The channel attention module uses global max pooling and 1×1 convolution to extract global and local channel attention respectively. The extracted parameters are then converted into feature weights using the Sigmoid function and multiplied by the input feature map.

[0020] X′=X×σ(f 1×1 (ReLU(f 1×1 (GMP(x))))+f 1×1 (ReLU(f 1×1 (X))))

[0021] GMP(·) is the global max pooling operation.

[0022] In the second step above, the optimization process of the model is controlled by the joint loss function. Specifically, the joint loss function consists of the segmentation loss, the classification loss, and a parameter balancing the two, and its form is:

[0023] L total =θ(1-λ)L Seg +δλL Cls

[0024] Where L Seg For the segmentation loss, L Cls For classification loss, θ and δ are the balance coefficients that balance the segmentation loss and the classification loss, and λ is a weighting factor controlled by the number of iterations, which is the ratio of the current iteration number to the total number of iterations.

[0025] Because the number of samples containing defects often constitutes only a small portion of the entire dataset during defect detection model training, and the number of positive pixels is often much smaller than the number of negative pixels in segmentation, positive samples are often difficult to classify during model training. Therefore, this patent uses FocalLoss as the model's segmentation loss, which takes the following form:

[0026] L Seg =-(1-p t ) γ log(p t )

[0027] Furthermore, due to the problem of unclear differences between categories and poor classification performance in some multi-class defect detection problems, this patent adopts Large Margin Cross-Entropy Loss as the classification loss. Its main function is to force the model to learn features with larger inter-class distances and smaller intra-class distances. The form is as follows:

[0028]

[0029] Where y is the label corresponding to the true category of the defect, and f is the classifier, f y f is the classification score corresponding to the true category. c Let ε be the classification score corresponding to category c, and ε be the regularization coefficient that constrains the dispersion of classification scores of non-target classes. Attached Figure Description

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] Figure 1 This is a schematic diagram of the structure of the multi-receptive field spatial attention module of the present invention;

[0032] Figure 2 This is a schematic diagram of the hybrid attention feature fusion module proposed in this invention, which combines the above two attention modules.

[0033] Figure 3 This is a schematic diagram of the overall structure of the defect detection model of the present invention.

[0034] Figure 4 This is a schematic diagram of the results output when this method is applied to the MAGNETIC-TILE dataset for defect detection. Detailed Implementation

[0035] Figure 1 The spatial attention module shown is an improvement on the SAM module in the CBAM model. Unlike CBAM, which directly uses convolutional kernels of different sizes for computation, the spatial attention module proposed in this invention expands the receptive field through dilated convolution. Based on a 3×3 convolutional kernel, different dilation rates are set to obtain different receptive fields. The use of dilated convolution not only better expands the receptive field, fully utilizes multi-scale contextual information, and enhances the ability to capture small defects, but also better controls the number of parameters, i.e., no additional parameters need to be introduced. Simultaneously, the features extracted using a standard convolutional kernel with a dilation rate of 1 effectively avoid the problems of local information loss and lack of correlation in the extracted information caused by dilated convolution.

[0036] Considering that positive pixels containing defects often only account for a small portion of the entire image in defect detection problems, extracting attention weights using global average pooling is difficult to achieve good results. Therefore, this invention's channel attention module uses global max pooling (GMP) to extract global channel attention. The input feature map of dimension H×W×C is transformed into a 1×1×C one-dimensional feature vector after GMP operation. After convolution with a kernel size of 1×1, the dimension changes to... (In this invention, r is set to 2). After activation by the ReLU function, it is restored to the initial dimension of 1×1×C by a 1×1 convolution operation. The main purpose of this operation is to speed up the operation and save training time. In addition, the input feature map only obtains local channel attention through the dimension scaling operation of 1×1 convolution. After the two are added together, they are activated by the Sigmoid function and transformed into channel attention weights of the input features. After multiplying with the input features, the weighted feature map is obtained.

[0037] Figure 2 The hybrid attention feature fusion module combines spatial and channel attention to construct an encoder-decoder network where shallow and deep features are fused using this hybrid attention feature fusion method. The encoder input feature X and the feature map Y from the lower layer of the decoder, after bilinear interpolation upsampling, are combined element-wise and input into the channel attention module to obtain the channel-oriented attention weights. These weights are then multiplied by X and Y respectively, and the resulting feature maps are used as inputs to two multi-receptive-field spatial attention modules. The spatial attention-weighted results are then concatenated to produce the output feature map of the entire hybrid attention feature fusion module, which continues to participate in the computation of the next higher layer of the network. The arrangement of the two attention types follows the sequential arrangement of the CBAM module—channel domain first, then spatial domain—significantly improving the model's feature extraction capability.

[0038] Figure 3 This is a schematic diagram of the overall structure of the defect detection model in this patent. The backbone of the model is an encoder-decoder structure. The input image is first downsampled four times using max pooling to obtain a high-dimensional feature map of 64×64×256. An additional multi-receptive-field spatial attention module generates spatial attention weights for the high-dimensional features, which are multiplied by the input features of the classification branch, thus enhancing the features of the classification network. Before downsampling, each layer's feature map is input into a hybrid attention feature fusion module and fused with the upsampled feature map from the deeper layers below. This method can fully utilize global context information and improve the performance of the segmentation network. The H×W×1 segmentation image output by the segmentation network is pooled and concatenated with the output of the classification network to form a 66×1 feature vector, which is then input into a fully connected layer to generate probability scores for each category.

[0039] Example 1

[0040] This embodiment employs a defect detection method based on hybrid attention feature fusion and joint optimization to perform defect detection on the image to be detected. Here, this embodiment performs defect detection on the MAGNETIC-TILE dataset.

[0041] The MAGNETIC-TILE dataset contains 1344 images of five types of defects on automotive magnetic tile surfaces (porosity, breakage, cracks, wear, and unevenness) as well as a large number of defect-free samples. Among these, there are 392 defective samples, which include pixel-level annotations that match the shape of the defects.

[0042] The first step is to preprocess the original images in the dataset, dividing the data into training and test sets. The input images are converted to single-channel grayscale images using the `IMREAD_GRAYSCALE` method in the `opencv-python` library, then converted to tensors and standardized. Simultaneously, the labels corresponding to the training set samples are read in, and mislabeled pixels with pixel values ​​between 0 and 1 in the segmentation labels are set to 1 with a threshold of 0.5. This is done to avoid the potential problem of low confidence in defect boundary segmentation due to mislabeling of defect edges in the dataset's segmentation labels.

[0043] The second step involves inputting the preprocessed image into the network model. The initial input is a 512×512 single-channel grayscale image with a batch size of 8. The data is first fed into the segmentation network, which consists of an encoder stage and a decoder stage. The encoder stage includes five convolutional modules and four downsampling operations. The convolutional modules include two 3×3 kernel convolutions that double the number of output channels, two normalization (BatchNorm2d) operations, and two ReLU operations. The normalization operation standardizes the data based on the mean and variance of the input data, making the data more statistically meaningful and preventing the data from becoming too large and affecting the network's stability before the ReLU operation. The ReLU operation effectively solves the problems of slow backpropagation and high computational cost of activation functions such as sigmoid, effectively saving computation time and avoiding problems such as gradient vanishing and overfitting.

[0044] The decoder stage consists of four upsampling-feature fusion-convolution operations. The upsampling operation consists of an UpSample operation with a stride of 2, a 3×3 convolution operation, and a combination of normalization and ReLU operations. The UpSample operation restores the image size to twice the input size, and the convolution operation compresses the number of image channels to half of the original number. The image is then fused with the downsampled image from the encoder stage through long-skip connections into the hybrid attention feature fusion module.

[0045] The input to the hybrid attention feature fusion module consists of two parts: the recovered feature map X after upsampling in the decoder stage and the feature map Y of the corresponding layer in the encoder stage. The two sets of input feature maps are first summed along the channel direction. The accumulated feature map is then processed by the channel attention module to calculate local attention (calculated through global max pooling) and global attention (calculated through convolution). The two attention weights are added together, activated by the sigmoid function, and multiplied by the accumulated input feature to obtain the total channel attention weight. This weight is then multiplied by X and Y respectively. The resulting weighted features are then spatially weighted by the multi-receptive field spatial attention module and concatenated along the channel direction to produce the final hybrid attention fusion feature, expressed by the following formula:

[0046] z = concat[(f s (X×f c (X+Y)); f s (Y×f c (X+Y))]

[0047] Where Z represents the output of the hybrid attention feature fusion module, concat[·;·] represents the concatenation operation along the channel direction, and f s (·) indicates a multi-receptive-field spatial attention module, f c (·) represents the channel attention module, whose expressions are as follows:

[0048]

[0049]

[0050] Wherein, GMP(·) is the global max pooling operation, used to transform the feature map into a one-dimensional feature vector along the channel direction; f 1×1 (·) represents a convolution operation with a kernel size of 1×1, mainly used to compress the number of channels. σ(·) represents the Sigmoid activation function, which normalizes the attention weights to between 0 and 1. This indicates that a 3×3 convolution kernel based on dilated convolution with different dilation rates can effectively extract image features under different receptive fields. When d=1, it is a standard 3×3 convolution operation. This method can effectively avoid the problem of missing small features that may be caused by dilated convolution while improving the model's receptive field.

[0051] After four upsampling reconstruction operations in the decoder stage, the deep features of the segmentation network output a pixel-level segmentation map of the input image. This segmentation map has the same size as the input image and can indicate the shape and location of defects in the defect image. This segmentation map is then flattened by the view function and concatenated with the feature vector output by the classification network after global average pooling and global max pooling, providing guidance for defect classification.

[0052] In the segmentation network, a 32×32×256 high-dimensional feature map generated by four downsampling operations in the encoder stage is used as input to the classification network. This network includes a multi-receptive-field spatial attention module, whose main function is to enhance the model's deep features and improve its ability to capture minute defects. This deep feature map is transformed into a 32×1 feature vector through three 5×5 convolution operations. This feature vector is then concatenated with the output of the segmentation network after global average pooling and global max pooling. The resulting 66×1 feature vector is input into a fully connected layer with an output of 6. After a softmax operation, it is transformed into probability scores for the six corresponding classes, the sum of which is 1. An acceptance threshold of 0.5 is set, meaning that the classification result is considered acceptable when the probability of the class with the highest score among the six classes exceeds 0.5.

[0053] In the third step, the outputs of the segmentation network and the classification network are used to calculate the loss against their corresponding ground truth labels. The ground truth label for the segmentation network is a black-and-white binary image of the defect image, with defective pixels labeled as 1 and non-defective pixels labeled as 0. Binary classification is performed for each pixel. The label for the classification network is the number of defect categories plus a unique identifier for the non-defective category, and the loss value is calculated using the following loss function:

[0054]

[0055] The segmentation loss is calculated by averaging the losses of each pixel in the image. The gradient is then calculated using the backpropagation algorithm to optimize the model parameters.

[0056] The training environment for this method was built using Python 3.8 and PyTorch 1.7.1, with a training set to test set ratio of 4:1. The Adadelta algorithm was used for model optimization, with a learning rate of 0.1. The hardware environment used for model training consisted of an Ubuntu 18.04 operating system, an Intel 6140 CPU, and an Nvidia RTX 3090 GPU. The hyperparameters θ and δ in the joint loss function were set to 10 and 0.1, respectively. The value of γ in the segmentation loss was set to 2, and the hyperparameter ε in the classification loss was set to 0.3. The total number of training iterations was 300.

[0057] To evaluate the effectiveness of the proposed method in improving the performance of defect detection tasks, this method selects some outstanding algorithms and compares their performance with that of the model in this method under the same experimental conditions. To test the stability of the model's experimental results, each experiment is conducted three times under the same conditions, and the standard deviation is calculated. As shown in Table 1, the proposed method achieves the best results in all indicators.

[0058] Table 1. Comparison of experimental results for different algorithms

[0059]

[0060] The evaluation metrics in the table are precision, recall, F1-Measure, and accuracy, and their formats are as follows:

[0061]

[0062]

[0063]

[0064]

[0065] Where c represents the number of categories, and all indicators are calculated using a macro average method. The advantage of this method is that it can avoid the adverse effects caused by excessive differences in the number of samples in each category.

[0066] Figure 4 The figure provides a visual representation of the model's segmentation network output. Columns 1 to 5 correspond to five types of samples: unevenness, damage, porosity, cracks, and wear. The figure demonstrates that, when faced with a dataset with pixel-level annotations, the proposed model can accurately segment pixels containing defects, and it can still accurately extract defect information even when there are multiple defect categories with significant differences in defect types.

[0067] The segmentation and classification sub-loss functions involved in this invention, as well as the channel attention implementation method in the hybrid attention feature fusion module, are all improvements based on existing methods.

[0068] It should be noted that this invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. A defect detection method based on joint optimization and hybrid attention feature fusion, characterized in that: The steps of this method are as follows: Step 1: Acquire images of the surface of the workpiece to be tested, preprocess the acquired images, set the real labels used for training, and build a network model. The model is divided into two parts: a segmentation network and a classification network. The segmentation network consists of an encoder stage, a decoder, and a hybrid attention feature fusion module. The classification network consists of a convolutional network backbone, a multi-receptive field spatial attention module, and a classifier. Step 2: Input the preprocessed image into the model for training, construct the joint loss function, set the optimization parameters and the number of iterations, and the model output is the pixel-level segmentation map of the defect area and the corresponding defect type; Step 3: Save the trained model weights and use the model to detect surface defects on the workpiece. A defect detection model based on hybrid attention feature fusion is constructed by a segmentation network with an encoder-decoder structure and a classification network that incorporates a multi-receptive-field spatial attention module. The segmentation network consists of an encoder-decoder backbone and a hybrid attention feature fusion module. The encoder part includes four consecutive downsampling operations with a stride of 2. The feature maps extracted by each layer are input to the hybrid attention feature fusion module, and are then stitched together with the feature maps of the same resolution after reconstruction by the decoder structure before participating in subsequent convolution calculations. Finally, the decoder outputs a pixel-level segmentation map with the same size as the input image, indicating the location and shape of defects. The deep features output by the encoder stage of the segmentation network are used as input to the classification network, and are then weighted by the multi-receptive field spatial attention module and subjected to convolution operations. Spatial attention module based on multiple receptive fields and hybrid attention feature fusion module The hybrid attention feature fusion module combines spatial and channel attention to construct an encoder-decoder network. Shallow and deep features are fused using this hybrid attention feature fusion method, and the encoder input features... The feature map of the lower layer of the decoder after bilinear interpolation upsampling operation After element-wise addition, the results are input into the channel attention module to obtain the attention weights for each channel direction. These weights are then compared with... , The feature map generated after multiplication is used as the input of two multi-receptive-field spatial attention modules. The result after spatial attention weighting is concatenated to generate the output feature map of the entire hybrid attention feature fusion module, which continues to participate in the calculation of the next higher layer network. The hybrid attention feature fusion module consists of a multi-receptive field spatial attention module and a channel attention module. The multi-receptive field spatial attention module has a dimension of... The input feature map X is convolved under different receptive fields, activated by the non-linear activation function ReLU, and then concatenated. Feature maps, and then... Convolution compresses the number of channels in the feature map to 1. After activation by the Sigmoid function, it is multiplied by the original input feature map to obtain the attention-weighted feature map. The channel attention module employs global max pooling and The convolution operation extracts global and local channel attention respectively, and the extracted parameters are converted into feature weights by the Sigmoid function, and then multiplied with the input feature map to obtain... .

2. The defect detection method based on joint optimization and hybrid attention feature fusion according to claim 1, characterized in that: in This indicates a splicing operation. Indicated by Based on convolution kernels of varying sizes, convolution operations are performed using different receptive fields. Represents the ReLU activation function; in This is a global max pooling operation.

3. The defect detection method based on joint optimization and hybrid attention feature fusion according to claim 1, characterized in that: The third step is characterized by an optimization method for a segmentation-classification two-stage defect detection model based on joint optimization. The core of this optimization method is the constructed joint loss function, which takes the following form: in and To balance the segmentation loss and classification loss, This is a weighting factor controlled by the number of iteration rounds, expressed as the ratio of the current iteration round to the total number of iteration rounds. This is a modulation factor indicating the ease or difficulty of sample segmentation. For classifiers, The classification score corresponding to the true category. Let be the classification score corresponding to category c, and 𝜀 be the regularization coefficient that constrains the dispersion of classification scores for non-target classes.

Images