Workpiece surface defect detection method based on multi-attention mechanism
By constructing a neural network model that integrates multiple attention mechanisms, the problems of low efficiency and poor robustness of traditional detection methods are solved, achieving high-precision and efficient workpiece surface defect detection. It is applicable to a variety of materials and defect types, and improves the intelligence and automation level of the production line.
Patent Information
- Application Number
- CN202411119182.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-15
AI Technical Summary
Traditional methods for detecting surface defects in workpieces rely on manual inspection and conventional image processing techniques, which suffer from low efficiency and poor robustness, especially when dealing with defects of multiple scales and complex textures.
We employ a deep learning approach based on multi-attention mechanisms to construct a neural network model that integrates these mechanisms. This model improves detection accuracy and efficiency through multi-layered and multi-angle attention mechanisms, including data preprocessing, a multi-attention mechanism RPN network, and ROI Align technology, thereby enabling automatic feature extraction and defect detection.
It improves the accuracy and efficiency of workpiece surface defect detection, enhances the robustness of the model, is applicable to various materials and defect types, reduces production costs, and improves the intelligence and automation level of the production line.
Smart Images

Figure CN118967644A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image surface defect detection, and in particular relates to a workpiece surface defect detection method based on a multi-attention mechanism. Background Art
[0002] With the rapid development of China's manufacturing industry, the number and types of industrial products are increasing day by day. Consumers and manufacturers have put forward higher requirements for product quality. In the industrial manufacturing process, the surface quality of the workpiece directly affects the performance and life of the product. Therefore, workpiece surface defect detection has always been an important link in industrial production. Traditional workpiece surface defect detection methods mainly rely on manual detection and traditional image processing technology, but these methods have many limitations. With the rapid development of deep learning technology, defect detection methods based on convolutional neural networks have gradually become a research hotspot. Deep learning methods can automatically extract image features and have strong robustness and generalization capabilities, showing excellent performance in many visual tasks. However, traditional convolutional neural networks still have some challenges when dealing with multi-scale and complex texture defect detection problems. As a method that can dynamically adjust the model's attention area, the attention mechanism has achieved remarkable results in the field of computer vision in recent years. The attention mechanism can assign different weights according to the importance of different parts of the input data, thereby improving the detection accuracy and robustness of the model. Summary of the invention
[0003] The present invention provides a workpiece surface defect detection method based on a multi-attention mechanism, aiming to utilize a multi-level and multi-angle attention mechanism to effectively improve the detection accuracy and efficiency of workpiece surface defects.
[0004] In order to achieve the above object, the present invention provides the following scheme: a method for detecting surface defects of a workpiece based on a multi-attention mechanism, comprising the following steps:
[0005] Step 1) Obtain industrial product defect image data and annotate it to obtain a labeled data set;
[0006] Step 2) preprocessing the data and dividing the data set;
[0007] Step 3) construct a neural network model integrating multiple attention mechanisms, determine the loss function, take the defect image in the data set as input and the label data as output, and use the reverse conduction method to perform iterative fitting prediction training to obtain an ideal model of the defect detection network integrating multiple attention mechanisms with enhanced weights on defect parts;
[0008] Step 4) Use the ideal model to perform classification prediction and regression prediction on the image of the workpiece to be inspected, and output the defect category and defect location.
[0009] It uses industrial cameras to collect surface defect images of workpieces and summarizes them into industrial product defect image datasets; uses the labeling tool LabelImg to label the defect locations, including marking location boxes and labeling defect types, to obtain a labeled dataset.
[0010] The data preprocessing includes: amplifying and expanding the samples by cutting, moving, flipping and changing the brightness; the divided data set is randomly divided in proportion, and each set contains all defect types.
[0011] The neural network model that integrates multiple attention mechanisms includes:
[0012] Step 31) The input image is proportionally resized and input into a feature extraction network with ResNet-50 as the backbone to obtain the feature map F1 of the image, which is used for classification and regression of the subsequent generation network;
[0013] Step 32) The feature map F1 generated in step 31) is used as the input of the RPN network fused with the multi-attention mechanism. The RPN network fused with the multi-attention mechanism identifies and locates the defect and outputs the position candidate box. For some candidate boxes that may overlap, the non-maximum suppression NMS algorithm is used to screen the candidate boxes, and the bounding box regression is used to correct the eliminated anchor boxes and obtain the final suggestion box.
[0014] Step 33) divides the proposed box in step 32) into grids of equal size, and then performs a pooling operation in each grid to aggregate the feature information in the grid into a fixed size, and outputs a feature map F2 for subsequent classification and bounding box regression;
[0015] Step 34) inputs the feature map F2 obtained in step 33) into the fully connected layer and the Softmax classifier to calculate the probability that the proposed box contains the target. The fully connected layer converts the feature map F2 into a one-dimensional vector and outputs the probability distribution of each category through the Softmax classifier to determine whether the proposed box contains the target; at the same time, the bounding box regression is used to fine-tune the position of the proposed box according to the difference between the proposed box and the real box of its corresponding label, so that the final target detection box is more accurate.
[0016] The iterative fitting prediction training includes: during the iterative training process, the total loss function is calculated to converge the network parameters, and the network parameters are updated in reverse to obtain an ideal model.
[0017] The RPN network integrating multiple attention mechanisms in step 32) includes: performing 3×3 convolution on the input feature map to obtain feature F; obtaining a more refined feature F″ along the multi-convolution attention module of the feature map F; inputting the obtained feature map F″ into the RPN network for subsequent more accurate target classification and regression; and being used to improve the model's attention to defect features and suppress features of complex backgrounds and pseudo-defects, thereby guiding the model to more accurately predict the location of defects under the interference of complex backgrounds.
[0018] The process of the RPN network processing the input feature map F″ in the step 32) is as follows: for the input feature map F″, a sliding window operation is performed on the feature map F″ using a sliding window of a fixed size, and multiple anchor boxes Anchor of different sizes and proportions are generated with each position as the center point; the upper branch uses Softmax classification to classify the anchor boxes into positive and negative samples, and judges each wrong box as a positive classification that may contain the target or a negative classification that does not contain the target; the lower branch is used to perform bounding box regression Bounding Box Regression, which calculates the offset between the predicted bounding box and the real target bounding box of the corresponding label to obtain a more accurate candidate box position.
[0019] In the step 32), for some candidate boxes that may overlap, the non-maximum suppression NMS algorithm is used to screen the candidate boxes, and the bounding box regression is used to correct the eliminated anchor boxes to obtain the final suggestion boxes, including: during the model testing process, for some candidate boxes that may overlap, the non-maximum suppression NMS algorithm is used to screen these boxes, and representative boxes are selected as output and used as the region of interest ROI; during the model training process, the Smooth_L1_Loss function is used to balance the influence between small differences and large differences, and the difference between the suggestion box and the real box is calculated to adjust the position of the suggestion box; then the position of each anchor box in the original image is determined, and it is further determined whether it exceeds the boundary of the original image. When the anchor box exceeds the boundary, it is determined that it does not contain target information, it is invalid and needs to be eliminated, and then the bounding box regression is used to correct the eliminated anchor box to obtain the final suggestion box.
[0020] The multi-convolutional attention module is composed of two complementary modules: channel attention and spatial attention, which are used to suppress the features of complex backgrounds while highlighting the features of defects.
[0021] Assume that the input feature map is F∈R C*H*W ,First, the attention map M is obtained through the channel attention module c ∈R C*1*1 , M cEach channel of the input feature F is weighted, and then the refined feature F′ is obtained, and then the attention map M is obtained through the spatial attention module s ∈R 1*H*W , M s (F′) is multiplied by the feature F′ to obtain the final output F″, which is expressed as follows:
[0022]
[0023]
[0024] in, Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, and H and W represent the height and width of the feature map.
[0025] For the final feature map, ROI Align introduces the idea of linear interpolation to integrate the feature aggregation process into a whole, avoiding quantization and rounding operations, thereby obtaining more accurate location information. At the same time, the target classification and location regression module uses the region proposal network to generate defect candidate boxes and uses the fully connected layer to determine their positions and categories to obtain the final detection results.
[0026] The present invention has the following beneficial effects and advantages:
[0027] The present invention provides a workpiece surface defect detection method based on a multi-attention mechanism, which improves detection accuracy and efficiency by integrating the multi-attention mechanism; the end-to-end deep learning model realizes automatic feature extraction and defect detection, reduces manual intervention, and improves detection efficiency and consistency; multi-scale feature extraction and adaptive feature weighting enhance the robustness of the model, and can cope with complex backgrounds and multiple types of defects; the method has strong versatility, is applicable to a variety of materials and defect types, reduces production costs, improves production efficiency, has broad application prospects and scalability, can be integrated into existing production line detection systems, and improves the intelligence and automation level of industrial production. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is a flow chart of a workpiece surface defect detection method based on a multi-attention mechanism;
[0029] Figure 2 It is an RPN network that integrates multiple attention mechanisms;
[0030] Figure 3 Schematic diagram of the process from RPN generating candidate boxes to outputting the final detection results. DETAILED DESCRIPTION
[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.
[0032] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0033] like Figure 1 The flowchart of the workpiece surface defect detection method based on the multi-attention mechanism shown includes the following steps:
[0034] First, the industrial product data set is acquired and preprocessed. In the above workpiece surface defect detection method, the steps for acquiring the industrial product data set are as follows:
[0035] Using industrial cameras, we can obtain images of workpiece surface defects, summarize image data, and obtain industrial product defect datasets. At the same time, we use the labeling tool LabelImg to label the collected workpiece surface images and obtain labeled training data.
[0036] The implementation process of preprocessing the data set in the above workpiece surface defect detection method is as follows:
[0037] Since a large number of data samples are required to avoid overfitting when training the network model, the existing data set is augmented. The samples are amplified using four methods that are closest to real industrial product images: cropping, position movement, flipping, and changing brightness. Finally, the data set is obtained. The acquired data set is randomly divided into training set, validation set, and test set images in a ratio of 8:1:1 according to the defect type. The training set data is used for training, the validation set is used to verify the model accuracy, and the test set data is used as the data to be tested.
[0038] The detection network model constructed in the embodiment refers to Figure 2-Figure 3 After obtaining the industrial product data set, we need to build a neural network model integrating multiple attention mechanisms. The model mainly includes a feature extraction module, an RPN network module integrating multiple attention mechanisms, and a target classification and regression module. The method for building a neural network model integrating multiple attention mechanisms includes the following sub-steps:
[0039] 1) Using ResNet-50 as the backbone network for feature extraction, the input image is resized proportionally and then input into the feature extraction network to obtain the feature map of the image for subsequent classification and regression of the generative network.
[0040] 2) The feature map generated in step 1) is used as the input of the RPN fused with the multi-attention mechanism. For some candidate boxes that may overlap, the non-maximum suppression (NMS) algorithm is used to filter these boxes, and the bounding box regression is used to correct the eliminated anchor boxes and obtain the final proposal box.
[0041] 3) Divide the proposed box input in step 2) into grids of equal size, and then perform a pooling operation within each grid to aggregate the feature information within the grid into a fixed-size output for subsequent classification and bounding box regression.
[0042] 4) Input the feature map obtained in step 3) into the fully connected layer and Softmax classifier to calculate the probability that the proposed box contains the target. The fully connected layer converts the feature map into a one-dimensional vector. The Softmax classifier outputs the probability distribution of each category to determine whether the proposed box contains the target. At the same time, the bounding box regression is used to fine-tune the position of the proposed box according to the difference between the proposed box and its corresponding real box, so that the final target detection box is more accurate.
[0043] 5) Calculate the total loss function, update the network parameters, and obtain a workpiece surface defect detection model based on the attention mechanism;
[0044] 6) Input the surface image of the workpiece to be inspected into the workpiece surface defect detection model, and finally output the detection result.
[0045] In the above-mentioned workpiece surface defect detection method, the implementation process of the RPN network integrating multiple attention mechanisms is as follows: In order to improve the model's attention to defect features and suppress the features of complex backgrounds and pseudo-defects, the RPN network integrating multiple attention mechanisms is adopted to guide the model to more accurately predict the location of defects under the interference of complex backgrounds. The specific process is to first perform a 3×3 convolution on the input feature map to obtain the feature F. After that, the feature map obtains a more refined feature F″ along the convolutional attention module, and finally the obtained feature map is input into the RPN network for subsequent more accurate target classification and regression.
[0046] The detection network model constructed in the embodiment refers to Figure 3 , where the main process of RPN network processing input feature map is as follows:
[0047] For the input feature map, a sliding window of a fixed size is used to perform a sliding window operation on the feature map, and multiple anchor boxes of different sizes and proportions are generated with each position as the center point. The upper branch uses Softmax classification to classify the anchor box into positive and negative samples, and judges each wrong box as a positive classification that may contain the target or a negative classification that does not contain the target; the lower branch is used for bounding box regression (Bounding Box Regression). This regression operation calculates the offset between the predicted bounding box and the true target bounding box to obtain a more accurate candidate box position. During the model testing process, for some candidate boxes that may overlap, the non-maximum suppression (NMS) algorithm is used to screen these boxes, and representative boxes are selected as output and used as the region of interest (ROI). During the training of the model, the Smooth_L1_Loss function is used to balance the influence of small differences and large differences, and the difference between the proposed box and the true box is calculated to adjust the position of the proposed box. Then, the position of each anchor box in the original image is determined, and it is further determined whether it exceeds the boundary of the original image. When the anchor box seriously exceeds the boundary, these anchor boxes basically do not contain target information, are invalid and need to be excluded, and then the bounding box regression is used to correct the eliminated anchor box and obtain the final proposed box.
[0048] The detection network model constructed in the embodiment refers to Figure 2 , the implementation process of the multi-attention module in the above RPN network integrating the multi-attention mechanism is as follows:
[0049] The convolutional attention module is composed of two complementary modules: channel attention and spatial attention, which can suppress the features of complex backgrounds while highlighting the features of defects.
[0050] Assume that the input feature map is F∈R C*H*W ,First, the attention map M is obtained through the channel attention module c ∈R C*1*1 , M c Each channel of the input feature F is weighted, and then the refined feature F′ is obtained. Then the attention map M is obtained through the spatial attention module. s ∈R 1*H*W , M s (F′) is multiplied by the feature F′ to obtain the final output F″, which is expressed as follows:
[0051]
[0052]
[0053] in: Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, and H and W represent the height and width of the feature map.
[0054] For the channel attention module, the feature map is first average pooled (avgpooling, represented by Pavg in the formula) and max pooled (maxpooling, represented by Pmax in the formula) to extract two feature descriptions: and Secondly, these two different features are sent to the middle shared network layer respectively, and the shared network is applied to and After the features are obtained, the corresponding elements of the two features are added together, and then the channel attention map M is obtained through the sigmoid activation function. c ∈R C*1*1 The calculation formula is as follows:
[0055] M c (F) = σ(P MLP (Pavg(F))+P MLP (Pmax(F)))
[0056] Among them: The shared network layer represents a multi-layer perceptron (multi-layer perceptron, MLP, in the formula P MLP denoted by ), a neural network with two layers. σ represents the sigmoid activation function.
[0057] The input of spatial attention is the output F′ of the channel attention module, and the feature map is obtained by average pooling and maximum pooling. and After merging the feature maps, a 7×7 convolution kernel is used and a sigmoid activation function is used to obtain a new spatial attention map M. s The calculation formula for spatial attention is:
[0058] M s (F) = σ(f 7*7 ([Pavg(F);Pmax(F)]))
[0059] Where: σ represents the sigmoid activation function, f 7*7 represents a convolution operation with a convolution kernel size of 7×7. Finally, M s (F′) is multiplied with its input features to obtain the final feature map F″.
[0060] For the final feature map, ROI Align introduces the idea of linear interpolation to integrate the feature aggregation process into a whole, avoiding quantization and rounding operations, thereby obtaining more accurate location information. At the same time, the target classification and location regression module uses the region proposal network to generate defect candidate boxes and uses the fully connected layer to determine their positions and categories to obtain the final detection results.
[0061] The above is a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A workpiece surface defect detection method based on multi-attention mechanism, It is characterized in that The steps include: Step 1) Obtain industrial product defect image data and annotate it to obtain a labeled data set; Step 2) Data preprocessing and data set division; Step 3) construct a neural network model integrating multiple attention mechanisms, determine the loss function, take the defect image in the data set as input and the label data as output, and use the reverse conduction method to perform iterative fitting prediction training to obtain an ideal model of the defect detection network integrating multiple attention mechanisms with enhanced weights on defect parts; Step 4) Use the ideal model to perform classification prediction and regression prediction on the image of the workpiece to be inspected, and output the defect category and defect location.
2. According to claim 1, a workpiece surface defect detection method based on a multi-attention mechanism, It is characterized in that It uses industrial cameras to collect surface defect images of workpieces and summarizes them into industrial product defect image datasets; uses the labeling tool LabelImg to label the defect locations, including marking location boxes and labeling defect types, to obtain a labeled dataset.
3. According to the method for detecting workpiece surface defects based on a multi-attention mechanism as described in claim 1, It is characterized in that The data preprocessing includes: amplifying and expanding the samples by cutting, moving, flipping and changing the brightness; the divided data set is randomly divided in proportion, and each set contains all defect types.
4. According to claim 1, a workpiece surface defect detection method based on a multi-attention mechanism, It is characterized in that The neural network model that integrates multiple attention mechanisms includes: Step 31) The input image is proportionally resized and input into a feature extraction network with ResNet-50 as the backbone to obtain the feature map F1 of the image, which is used for classification and regression of the subsequent generation network; Step 32) The feature map F1 generated in step 31) is used as the input of the RPN network fused with the multi-attention mechanism. The RPN network fused with the multi-attention mechanism identifies and locates the defect and outputs the position candidate box. For some candidate boxes that may overlap, the non-maximum suppression NMS algorithm is used to screen the candidate boxes, and the bounding box regression is used to correct the eliminated anchor boxes and obtain the final suggestion box. Step 33) divides the proposed box in step 32) into grids of equal size, and then performs a pooling operation in each grid to aggregate the feature information in the grid into a fixed size, and outputs a feature map F2 for subsequent classification and bounding box regression; Step 34) inputs the feature map F2 obtained in step 33) into the fully connected layer and the Softmax classifier to calculate the probability that the proposed box contains the target. The fully connected layer converts the feature map F2 into a one-dimensional vector and outputs the probability distribution of each category through the Softmax classifier to determine whether the proposed box contains the target; at the same time, the bounding box regression is used to fine-tune the position of the proposed box according to the difference between the proposed box and the real box of its corresponding label, so that the final target detection box is more accurate.
5. According to claim 1, a workpiece surface defect detection method based on a multi-attention mechanism, It is characterized in that The iterative fitting prediction training includes: during the iterative training process, the total loss function is calculated to converge the network parameters, and the network parameters are updated in reverse to obtain an ideal model.
6. According to claim 4, a method for detecting workpiece surface defects based on a multi-attention mechanism, It is characterized in that The RPN network integrating multiple attention mechanisms in step 32) includes: performing 3×3 convolution on the input feature map to obtain feature F; obtaining a more refined feature F″ along the multi-convolution attention module of the feature map F; inputting the obtained feature map F″ into the RPN network for subsequent more accurate target classification and regression; and being used to improve the model's attention to defect features and suppress features of complex backgrounds and pseudo-defects, thereby guiding the model to more accurately predict the location of defects under the interference of complex backgrounds.
7. A workpiece surface defect detection method based on a multi-attention mechanism according to claim 6, It is characterized in that The process of the RPN network processing the input feature map F″ in the step 32) is as follows: for the input feature map F″, a sliding window operation is performed on the feature map F″ using a sliding window of a fixed size, and multiple anchor boxes Anchor of different sizes and proportions are generated with each position as the center point; the upper branch uses Softmax classification to classify the anchor boxes into positive and negative samples, and judges each wrong box as a positive classification that may contain the target or a negative classification that does not contain the target; the lower branch is used to perform bounding box regression Bounding Box Regression, which calculates the offset between the predicted bounding box and the real target bounding box of the corresponding label to obtain a more accurate candidate box position.
8. According to claim 4, a method for detecting workpiece surface defects based on a multi-attention mechanism, It is characterized in that In the step 32), for some candidate boxes that may overlap, the non-maximum suppression NMS algorithm is used to screen the candidate boxes, and the bounding box regression is used to correct the eliminated anchor boxes to obtain the final suggestion boxes, including: during the model testing process, for some candidate boxes that may overlap, the non-maximum suppression NMS algorithm is used to screen these boxes, and representative boxes are selected as output and used as the region of interest ROI; during the model training process, the Smooth_L1_Loss function is used to balance the influence between small differences and large differences, and the difference between the suggestion box and the real box is calculated to adjust the position of the suggestion box; then the position of each anchor box in the original image is determined, and it is further determined whether it exceeds the boundary of the original image. When the anchor box exceeds the boundary, it is determined that it does not contain target information, it is invalid and needs to be eliminated, and then the bounding box regression is used to correct the eliminated anchor box to obtain the final suggestion box.
9. The method for detecting workpiece surface defects based on a multi-attention mechanism according to claim 6, It is characterized in that The multi-convolutional attention module is composed of two complementary modules: channel attention and spatial attention, which are used to suppress the features of complex backgrounds while highlighting the features of defects. Assume that the input feature map is F∈R C*H*W ,First, the attention map M is obtained through the channel attention module c ∈R C*1*1 , M c Each channel of the input feature F is weighted, and then the refined feature F′ is obtained, and then the attention map M is obtained through the spatial attention module s ∈R 1*H*W , M s (F′) is multiplied by the feature F′ to obtain the final output F″, which is expressed as follows: in, Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, and H and W represent the height and width of the feature map.
10. The method for detecting workpiece surface defects based on a multi-attention mechanism according to claim 4, It is characterized in that For the final feature map, ROI Align introduces the idea of linear interpolation to integrate the feature aggregation process into a whole, avoiding quantization and rounding operations, thereby obtaining more accurate location information. At the same time, the target classification and location regression module uses the region proposal network to generate defect candidate boxes and uses the fully connected layer to determine their positions and categories to obtain the final detection results.
Citation Information
Cited By
Electronic device detection system based on computer vision
CN119784754A
Marine food management method based on big data
CN119919005A
Target defect detection method and device
CN119991581A
Surface defect detection method and system based on deep learning
CN121032953A
Open set product surface fine defect detection method and device based on machine touch sense
CN121074022A
