A hierarchical classification defect detection method based on attention mechanism guidance

Abstract

The present application relates to a kind of layered classification defect detection method based on attention mechanism guidance, for the image texture background influence defect extraction problem in workpiece surface defect detection process, layered attention module is proposed, and integrated with the defect detection network of layered classification network structure, make model utilize image texture background guide defect detection, simultaneously, multi-receptive field fusion module is designed, utilize the enhanced receptive field brought by hollow convolution, make model better extract the defect features of different sizes, and reduce computational complexity, for the problem that background texture classification and defect classification training direction is difficult to grasp, a kind of joint optimization framework is proposed, utilize the joint loss function constructed to make model end-to-end training.Experiments show that the method proposed in the present application can effectively improve the detection accuracy of workpiece surface defect detection task.

Description

Technical Field

[0001] The technical solution of this invention relates to the field of workpiece surface defect detection using deep learning and convolutional neural network methods, specifically a hierarchical classification defect detection method based on attention mechanism guidance. Background Technology

[0002] In industrial production, certain uncontrollable factors can lead to surface defects on some workpieces, which can pose safety hazards to subsequent processing and use of the products. Defect detection technology is an indispensable and important means of quality control in modern industrial production.

[0003] Traditional manual methods for inspecting workpiece surface defects consume significant human, material, and time resources. Furthermore, manual processing is highly subjective and susceptible to various environmental factors. Additionally, the identification of small surface defects often results in missed or false detections, reducing the accuracy of the inspection.

[0004] In classic machine vision methods, researchers develop different detection methods for different defects, manually extracting defect features and then classifying them using classifiers. Traditional detection methods use hand-crafted features to model the differences between defective and non-defective regions, such as Scale Invariant Special Transform (SIFT), Histogram of Oriented Gradients (HOG), and Local Binary Pattern (LBP). Commonly used methods include statistical methods, structural methods, filter-based methods, and model-based methods. These methods are only designed and modeled for specific surfaces or defects, making it difficult to generalize to other types of surfaces and defects. Furthermore, many detection methods have extremely stringent requirements for the detection environment. Therefore, compared to manual detection methods, traditional machine vision methods can automatically detect defects, but their applicability is limited, their robustness is poor, and they face significant challenges in detecting complex defects.

[0005] With the deepening research in the field of deep learning, deep neural network models, represented by Convolutional Neural Networks (CNNs), have been widely applied in the field of defect detection, demonstrating outstanding performance in defect feature extraction and defect classification. The defect detection problem can be simply summarized as a binary classification problem of identifying whether an image to be detected contains a defect.

[0006] Most existing classification-based convolutional neural network (CNN) surface defect detection methods suffer from insufficient utilization of texture and background information. The main shortcomings of classification-based CNN surface defect detection methods are:

[0007] Traditional classification-based convolutional neural network surface defect detection methods only consider the extraction of defect features, neglecting the differences in surface texture and background, and ignoring the influence of texture and background on defect detection. Therefore, it is necessary to take into account the image background texture information and make full use of the relationship between background texture and defects.

[0008] (2) Traditional classification-based convolutional neural network surface defect detection usually uses stacked convolutional layers to expand the receptive field and increase the detection capability for large-sized defects. However, increasing the number of convolutional layers increases the computational complexity of the entire model and also loses information about small-sized defects. The multi-receptive field fusion module proposed in this invention can better extract information about defects of different sizes, improve the defect classification accuracy of the model, and also reduce the number of parameters and reduce the complexity of the entire model. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the technical problem to be solved by this invention is: to design a hierarchical attention module that can effectively utilize the background information of the defect surface texture and reduce the impact of background information on defect detection; at the same time, a multi-receptive field fusion module is designed to improve the model's ability to extract features of defects of different sizes; and to use a backbone-branch network for end-to-end training, which significantly improves the accuracy of the defect detection task.

[0010] The technical solution adopted by this invention to solve this technical problem is: to provide a hierarchical classification defect detection method based on an attention mechanism, the steps of which are as follows:

[0011] Step 1: Acquire images of the surface of the workpiece to be tested, preprocess the acquired images, set the real labels used for training, and construct the network model and joint loss function. The model consists of two parts: the backbone network and the branch network.

[0012] The acquired images are downsampled to a size of 512×512 pixels, and the original RGB three-channel images are converted to single-channel images (grayscale images). After standardization, they are input into the model. The classification network training requires labeling based on whether different image surfaces have defects; samples with defects are labeled as positive, and samples without defects are labeled as negative. Furthermore, corresponding category labels are assigned based on the texture background type of the workpiece surface to train the texture background classification network.

[0013] The entire network consists of a backbone network and two branch networks. The backbone network contains 5 convolutional modules and 5 global max pooling (GMP) operations, which are mainly used to extract image feature maps.

[0014] The input to the branch network is the output of the backbone network. The feature map contains two branch networks: one branch network is the image texture background classification network, which mainly distinguishes the image background type; the other branch network is the image defect classification network, which mainly distinguishes whether the image has defects.

[0015] Specifically, the image is input into the backbone network, which outputs a feature map. This feature map is then input into two branch networks. The first branch network obtains the feature map through convolution. The feature vectors are then input into a fully connected layer and processed by Softmax to obtain the confidence score for each image category. The second branch network consists of a hierarchical attention module and a multi-receptive field fusion module. The confidence score information output from the first branch network is input into the hierarchical attention module of the second branch network, and combined with the multi-receptive field fusion module of the second branch network, the image defect classification confidence score is output.

[0016] The hierarchical attention module is an improvement on the channel attention module. The image texture and background classification confidence scores output by the first-layer branch network are added as weights to each channel of the feature map in the second-layer branch network, resulting in attention-weighted feature maps. :

[0017] (1)

[0018] in The feature maps generated by the backbone network. Indicates a fully connected layer. The Dropout function prevents overfitting. Represents the ReLU activation function. This represents the confidence level of the image texture background classification obtained after the Softmax function operation.

[0019] The multi-receptive-field fusion module consists of three different branches. Each branch comprises a 1x1 convolutional kernel and two different 3x3 dilated convolutions. The 1x1 convolutional kernel captures small defect features, while the two dilated convolutions at different scales capture larger defect features. After convolution operations, each branch is activated by the non-linear activation function ReLU and then concatenated. The feature map obtained is: :

[0020] (2)

[0021] in This indicates a splicing operation. Indicates Convolution operation with kernel size, Indicates Based on convolution kernels of varying sizes, convolution operations are performed using different receptive fields. This represents the ReLU activation function.

[0022] The second step is to input the preprocessed image into the model for training, set the optimization parameters and the number of iterations, and the model outputs the texture background category and the presence or absence of defects.

[0023] The optimization process of the model is controlled by a joint loss function. Specifically, the joint loss function consists of a texture background classification loss, a defect classification loss, and a parameter balancing the two, and its form is:

[0024] (3)

[0025] in For texture background classification loss, Loss is classified as defect. and To balance the texture background classification loss and defect loss, this invention uses a balancing coefficient in the first 50 rounds. and The values ​​are 0.8 and 0.2 respectively, in the last 50 rounds. and They are 0.1 and 0.9 respectively. The weighting factor, controlled by the number of iteration rounds, has the following form:

[0026] (4)

[0027] in This is the current training round number. This refers to the total number of training epochs. The training process is dynamic, gradually shifting the training focus from image texture and background classification to image defect classification.

[0028] Since each layer of the hierarchical tree represents a classification task, to avoid the decrease in the learning rate of the mean squared error loss function during gradient descent when using the sigmoid function, this patent uses the cross-entropy loss function for both the first and second layers. The loss function is:

[0029] (5)

[0030] in For the input sample, The label corresponding to the actual category of the defect. For classifiers.

[0031] Step 3: Save the trained model weights and use the model to detect surface defects on the workpiece.

[0032] Experimental results demonstrate that adding a hierarchical guidance module can effectively improve the network's ability to identify subtle defects and increase the overall accuracy of the network in detecting defects.

[0033] This invention analyzes current surface defect detection algorithms based on convolutional neural networks and their existing problems, and proposes a surface defect detection algorithm based on attention-guided hierarchical networks. Experiments were conducted on DAGM industrial surface defect data. The experiments demonstrate that this method uses a hierarchical classification approach to simultaneously predict textured surfaces and defects, achieving excellent detection performance. The hierarchical attention module and the multi-receptive field fusion module effectively improve the accuracy of the classification network in identifying defects. Attached Figure Description

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] Figure 1 This is a schematic diagram of the structure of the hierarchical attention module of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of the multi-receptive field module proposed in this invention by combining dilated convolution.

[0037] Figure 3 This is a schematic diagram of the overall structure of the defect detection model of the present invention. Detailed Implementation

[0038] Figure 1 The hierarchical attention module shown is an improvement on the SENet channel attention module. Unlike SENet, which directly uses the feature map for convolution to obtain weights, the hierarchical attention module proposed in this invention classifies the background texture of the image with confidence. As a weighting factor, it is increased to On the feature map channel, the hierarchical attention module not only makes better use of image texture background information and distinguishes different types of defects, but also better controls the number of parameters, i.e., it does not require the introduction of additional parameters.

[0039] Figure 2The multi-receptive-field fusion module shown is an improvement on the ASPP module in the DeepLab model. Unlike CBAM, which directly uses convolutional kernels of different sizes for computation, this invention combines convolutions of different sizes, including one 1×1 convolutional kernel and two 3×3 convolutional kernels. The expansion of the multi-receptive-field fusion module proposed in this invention is achieved through dilated convolution. Based on a 3×3 convolutional kernel, different dilation rates are set to obtain different receptive fields. The use of dilated convolution not only better expands the receptive field, fully utilizes multi-scale contextual information, and enhances the ability to capture small defects, but also better controls the number of parameters, i.e., no additional parameters need to be introduced. Simultaneously, using features extracted by the 1×1 convolutional kernel for fusion effectively avoids the problems of local information loss and lack of correlation in the extracted information caused by dilated convolution.

[0040] Figure 3 This is a schematic diagram of the overall structure of the defect detection model in this patent. The backbone network structure of the model involves downsampling the input image through 5 convolutional modules and 4 global max pooling operations to obtain a dimension of [missing information]. The high-dimensional feature map is generated. The first branch network classifies image texture and background, mainly distinguishing the type of image background; the second branch network classifies image defects, mainly distinguishing whether the image has defects. Specifically, the image is input into the backbone network, which outputs a feature map. The feature map is then input into the two branch networks. The first branch network obtains the high-dimensional feature map through convolution. The feature vectors are then input into a fully connected layer and processed by Softmax to obtain the confidence score for each image category. The second branch network consists of a hierarchical attention module and a multi-receptive field fusion module. The confidence score information output from the first branch network is input into the hierarchical attention module of the second branch network, and combined with the multi-receptive field fusion module of the second branch network, the image defect classification confidence score is output.

[0041] Example 1

[0042] This embodiment employs a defect detection method based on hybrid attention feature fusion and joint optimization to perform defect detection on the image to be detected. Here, this embodiment performs defect detection on the DAGM dataset.

[0043] The DAGM dataset contains 10 classes of workpiece texture background samples, including 10,000 defect-free samples and 1,500 defective samples, and includes image-level annotations for the images.

[0044] The first step is to preprocess the original images in the dataset by dividing the data into training and testing sets. The input images are converted to single-channel grayscale images using the `IMREAD_GRAYSCALE` method in the `opencv-python` library, then converted to tensors and standardized. Simultaneously, the labels corresponding to the training set samples are read in: samples without defects are labeled 0, and samples with defects are labeled 1.

[0045] The second step is to input the preprocessed image into the network model, with the initial input being... A single-channel grayscale image with a batch size of 8 is first input into the backbone network. The backbone network consists of 5 convolutional modules and 5 downsampling operations. Each convolutional module contains two kernels with a kernel size of [missing value]. The algorithm employs a convolution operation that doubles the number of output channels, two batch normalization (BatchNorm2d) operations, and two ReLU operations. The normalization operation standardizes the data based on the mean and variance of the input data, making the data more statistically meaningful and preventing the data from becoming too large before the ReLU operation, thus ensuring network stability. The ReLU operation effectively solves the problems of slow backpropagation and high computational cost associated with activation functions like sigmoid, significantly saving computation time and avoiding issues such as gradient vanishing and overfitting.

[0046] The branch network consists of two branches. The first branch is for image texture and background classification. The feature map output from the backbone network is input, passed through three fully connected layers, two ReLU activation functions, and two Dropout functions to obtain the image texture and background classification result, while also providing guidance for defect classification. The second branch is for image defect classification. The feature map output from the backbone network is input, passed through a hierarchical attention module and a multi-receptor field fusion module to obtain the image defect classification result.

[0047] The input to the hierarchical attention module mainly consists of two parts: the feature map output after the backbone network stage. The confidence score for the first layer image texture background classification is used. The input feature map is first compressed through convolutional blocks to the same number of channels as the number of background classifications. The confidence score information for the input texture background classification is added to each channel as weights, ensuring that each channel contains background information. The formula is as follows:

[0048] (6)

[0049] in The feature maps generated by the backbone network. Indicates Based on the size of the convolution kernel, Represents the ReLU activation function. This represents the confidence level of the image texture background classification obtained after the Softmax function operation.

[0050] The multi-receptive-field fusion module consists of three different branches. Each branch comprises a 1x1 convolutional kernel and two different 3x3 dilated convolutions. The 1x1 convolutional kernel captures small defect features, while the two dilated convolutions at different scales capture larger defect features. After convolution operations, each branch is activated by the non-linear activation function ReLU and then concatenated. The feature map obtained is: :

[0051] (7)

[0052] in This indicates a splicing operation. Indicates Convolution operation with kernel size, Indicates Based on convolution kernels of varying sizes, convolution operations are performed using different receptive fields. This represents the ReLU activation function. Add... Larger kernel sizes can effectively avoid the problem of missing small features that may be caused by dilated convolutions while increasing the receptive field of the model.

[0053] The first layer, image texture and background classification, goes through three fully connected layers, two ReLU activation functions, and two Dropout functions. The backbone network generates... The feature vectors are then processed using Softmax to transform them into probability scores for each of the 10 corresponding classes, with the sum of these scores being 1. An acceptance threshold of 0.5 is set, meaning that the classification result is considered acceptable when the probability of the class with the highest score among the 10 classes exceeds 0.5.

[0054] The second layer of image defect classification involves concatenating feature maps after passing through a hierarchical attention module and a multi-receptive field fusion module. The hierarchical attention module increases channel weights using texture background information, while the multi-receptive field fusion module extracts defect feature information at different scales. Finally, after a softmax operation, the results are converted into probability scores corresponding to the presence or absence of defects, with a sum of 1. An acceptance threshold of 0.5 is set, meaning that if the probability of a class exceeds 0.5, the classification result is considered acceptable.

[0055] In the third step, the outputs of the image texture / background classification network and the image defect classification network are used to calculate the loss against their corresponding ground truth labels. The ground truth labels for the texture / background classification network range from 0 to 9 (ten classes in total); the labels for the defect classification network are 0 and 1, with 1 for defective images and 0 for defect-free images. The loss value is calculated using the following loss function:

[0056] (8)

[0057] After calculating the loss value, the gradient is calculated using the backpropagation algorithm to optimize the model parameters.

[0058] The training environment for this method was built using Python 3.8 and PyTorch 1.7.1, with a training set to test set ratio of 4:1. The model was optimized using the Adadelta algorithm with a learning rate of 0.001. The hardware environment used for model training consisted of an Ubuntu 18.04 operating system, an Intel 6140 CPU, and an Nvidia RTX 3090 GPU. The total number of training iterations was 100.

[0059] To evaluate the effectiveness of the proposed method in improving the performance of defect detection tasks, this method selects some outstanding algorithms and compares their performance with that of the model in this method under the same experimental conditions. To test the stability of the model's experimental results, each experiment is conducted three times under the same conditions, and the average value is calculated. As shown in Table 1, the proposed method achieves the best results in all indicators.

[0060] Table 1. Comparison of experimental results for different algorithms (%)

[0061]

[0062] The evaluation metrics in the table are True Positive Rate (TPR), True Negative Rate (TNR), and Accuracy, and their formats are as follows:

[0063] (9)

[0064] (10)

[0065] (11)

[0066] Where c represents the number of categories, and all indicators are calculated using a macro average method. The advantage of this method is that it can avoid the adverse effects caused by excessive differences in the number of samples in each category.

[0067] The classification loss function, hierarchical attention module, and multi-receptive field fusion module implementation methods involved in this invention are all improvements based on existing methods.

[0068] It should be noted that this invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. A hierarchical classification defect detection method based on attention mechanism, characterized in that: Includes the following steps: Step 1: Acquire images of the workpiece surface to be tested, preprocess the acquired images, set the real labels used for training, and construct a network model. The model consists of two parts: a backbone network and branch networks. The backbone network is composed of a convolutional network, and the branch networks consist of two layers. The first layer of the branch network is an image texture and background classification network, which is composed of a classifier. The second layer of the branch network is an image defect classification network, which is composed of a hierarchical attention module, a multi-receptor field fusion module, and a classifier. Step 2: Input the preprocessed image into the model for training, construct the joint loss function, set the optimization parameters and the number of iterations, and the model outputs the image texture background type and whether the image contains defects. Step 3: Save the trained model weights and use the model to detect surface defects on the workpiece.

2. The hierarchical classification defect detection method based on attention mechanism as described in claim 1, characterized in that: By using an attention mechanism, image background texture information is added to the image defect feature map as weights. This, combined with a multi-receptive field fusion module, constitutes a hierarchical classification-based defect detection model. The backbone network consists of 5 convolutional modules and 5 global max pooling (GMP) operations, which are mainly used to extract image feature maps. The branch network consists of two layers. The first branch network consists of a fully connected layer, a ReLU activation function, and a Dropout function, which is used to distinguish the image texture background type. The second branch network consists of a hierarchical attention module and a multi-receptor field fusion module, which is mainly used to distinguish whether the image contains defects. The hierarchical attention module is an improvement on the SENet channel attention module. Unlike SENet, which directly uses the feature map itself for convolution to obtain weights, the hierarchical attention module uses image background texture information as weights added to the defect detection feature map channels. Specifically, the first-layer branch network calculates the image's background texture classification confidence. As a weight, it is added to the second-layer branch network. The feature map obtained from the feature map channel is : (1) in The feature maps generated by the backbone network. Indicates Based on the size of the convolution kernel, Represents the ReLU activation function. This represents the confidence level of the image texture background classification obtained after the Softmax function operation; Indicates a fully connected layer. This is to prevent overfitting; The multi-receptive-field fusion module consists of three different branches: one 1x1 convolutional kernel and two different 3x3 dilated convolutions. The 1x1 convolutional kernel captures small defect features, while the two dilated convolutions at different scales capture larger defect features. After convolution operations, each branch is activated by the non-linear activation function ReLU, and then concatenated. The feature map obtained is: : (2) in This indicates a splicing operation. Indicates Convolution operation with kernel size, Indicates Based on convolution kernels of varying sizes, convolution operations are performed using different receptive fields.

3. The hierarchical classification defect detection method based on attention mechanism as described in claim 1, characterized in that: In the third step, a joint loss function is constructed, which takes the following form: (3) For texture background classification loss, Loss is categorized by defect type; in The weighting factor is controlled by the number of iteration rounds, and is in the form of the current iteration round number. With the total number of rounds The ratio, and To balance the texture background classification loss and the defect classification loss, a balance coefficient is set. For texture background classifier, It is a defect classifier.

Images