A method for detecting surface defects in workpieces based on the YOLACT model

Abstract

This invention discloses a workpiece surface defect detection method based on the YOLACT model, relating to the fields of workpiece surface defect detection and machine vision technology. The method includes: S100, image acquisition; S200, image annotation; S300, data augmentation; S400, model construction; S500, model training; S600, model validation; and S700, defect detection. This invention exhibits strong robustness and generalization ability, improves the accuracy of defect target edge detection, can obtain finer-grained detection results, has higher detection speed, is more suitable for deployment on CPUs, and is more suitable for application in practical industrial production environments.

Description

Technical Field

[0001] This invention relates to the fields of workpiece surface defect detection and machine vision technology, and in particular to a workpiece surface defect detection method based on the YOLACT model. Background Technology

[0002] Workpieces are the most basic building blocks of industrial equipment, and their quality directly determines the overall performance of the product. Therefore, workpiece quality inspection is a crucial and indispensable part of industrial production. Thus, automating and intelligentizing the workpiece quality inspection process is imperative. However, in current workpiece production, manufacturers primarily rely on manual methods to inspect for defects, including visual observation and auditory inspection. This manual method obviously has many drawbacks, including high worker wages, slow inspection speed, subjective results due to individual worker differences, and errors caused by worker fatigue or boredom. In short, using manual methods to inspect workpiece defects increases production costs, makes it difficult to guarantee product quality consistency, reduces market competitiveness, and significantly impacts enterprise development.

[0003] To overcome the shortcomings of manual defect detection, workpiece manufacturers urgently need a system capable of automatically detecting product defects. Driven by this demand, various defect detection systems based on different technologies have been developed. Among them, automated defect detection technology based on digital image processing and machine vision is widely used in the field of defect detection due to its advantages such as high detection speed, low labor costs, and stable and reliable detection results. Because automated detection technology brings higher efficiency, it has been applied to the actual production processes of many industrial products, such as surface defect identification of steel strips, non-destructive testing of wood defects, fabric pattern defect detection, tile surface defect detection, electronic component shape defect detection, and LCD display defect detection. In the detection of surface defects in many products, due to the complex structure of the workpiece itself, traditional computer vision technology struggles to extract suitable feature vectors, resulting in poor detection performance. Therefore, methods using convolutional neural networks (CNNs) to automatically extract image features for defect detection have emerged. Compared to traditional computer vision methods, CNNs have the advantage of automatically extracting image features, avoiding the complex image feature extraction algorithm design process in image processing. A basic convolutional neural network (CNN) comprises multiple convolutional layers, downsampling layers, activation function layers, and fully connected layers, forming a hierarchical structure. The convolutional layer embodies local response theory and is used by the CNN to extract local features from images. The neural network fuses the extracted local features into global features through inter-layer connections and makes a final judgment, simulating the process of the human brain processing visual information. By designing a CNN with multiple convolutional layers, the network can process workpiece images with complex textures. Replacing traditional computer vision algorithms with this network as the system's image processing algorithm greatly improves the system's workpiece defect detection capability. In the field of workpiece surface defect detection in industrial scenarios, the most widely used is the YOLO (You Look Only Once) series of object detection network models that utilize convolutional neural networks to extract image features. This series of detection models can identify the defect category in an image and output the bounding box coordinates of the defect for defect localization. However, existing methods for workpiece surface defect detection in industrial scenarios still have the following problems:

[0004] 1. Currently, most workpiece surface defect detection methods implemented using convolutional neural networks are based on target detection models. Target detection methods can only obtain limited defect information, such as the bounding box of the defect target and its category, which is not conducive to further post-processing of the detection results and leads to low defect detection accuracy.

[0005] 2. Currently, most detection methods employ the SE-attention module to enhance the model's ability to extract image features. Specifically, the SE-attention module recalibrates the weights of different channels using global context, thereby adjusting channel dependencies. However, feature fusion methods using weight recalibration cannot fully utilize global contextual information; therefore, the method of using the SE-attention module to enhance the model's feature extraction capability requires further improvement.

[0006] 3. Currently, most detection methods only consider deployment on GPUs, so they basically only use ordinary convolutional modules in the model. However, ordinary convolutional modules are not very efficient on CPUs, which leads to a significant reduction in detection speed after the detection method is deployed on a CPU.

[0007] 4. Most current detection methods are derived from popular target detection models. Since the contrast between the defect target on the workpiece surface and the background is relatively low, these detection methods are not very effective at detecting the edge of the defect target.

[0008] Therefore, those skilled in the art are dedicated to developing a workpiece surface defect detection method based on the YOLACT model. Summary of the Invention

[0009] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to improve the accuracy of target edge detection of surface defects on workpieces, obtain more fine-grained detection results, achieve higher detection speed, be more suitable for deployment on CPU, and be more suitable for application in actual industrial production environments.

[0010] The inventors analyzed the problems existing in the current technology, conducted in-depth research on the factors affecting detection accuracy and speed, and proposed a lightweight detection method for workpiece surface defects in industrial scenarios. First, the YOLACT model is used as the baseline model for surface defect detection. The backbone network is a ResNet18 network with a GC-attention module added. Simultaneously, ordinary convolutional modules are replaced with depthwise separable convolutional modules to extract multi-scale feature maps from the workpiece surface image. Then, a BiFPN network is used to perform weighted multi-scale fusion of the input features extracted by the ResNet18 network. Finally, the detection head of the YOLACT model is used as the detection head. The multi-scale feature maps, after being fused by the BiFPN network, are input into the detection head for further processing to obtain the final defect detection result. To improve the edge detection effect of defect targets, an edge feature extraction module and an edge detection head are added to the model, running in parallel with the BiFPN network and the detection head.

[0011] In one embodiment of the present invention, a method for detecting surface defects in a workpiece based on the YOLACT model is provided, comprising:

[0012] S100, Image Acquisition: Uses an industrial camera to acquire images of the workpiece surface on an industrial production line;

[0013] S200, Image annotation: Use image annotation tools to annotate defects in the collected workpiece surface images and generate a workpiece surface defect dataset;

[0014] S300, Data Augmentation: Apply data augmentation strategies to the workpiece surface defect dataset to perform data augmentation.

[0015] S400, Model Building: Construct a workpiece surface defect detection model based on YOLACT;

[0016] S500, Model Training: The data-enhanced workpiece surface defect dataset is divided into training and testing sets according to a certain ratio. The training set is input into the workpiece surface defect detection model for training. The learning rate parameter is adjusted to make the loss function of the workpiece surface defect detection model converge, and a preliminary trained workpiece surface defect detection model is obtained.

[0017] S600, Model Validation: Defect detection is performed using the pre-trained workpiece surface defect detection model, i.e., defect prediction is performed using the pre-trained workpiece surface defect detection model to obtain a validated workpiece surface defect detection model.

[0018] S700, Defect Detection, deploys a validated workpiece surface defect detection model on industrial production lines to perform real-time workpiece surface defect detection.

[0019] Optionally, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, step S100 includes:

[0020] S110, Illumination highlights the surface defects of workpieces on industrial production lines;

[0021] S120. Use an industrial camera to capture images of the workpiece surface;

[0022] S130, Store workpiece surface image.

[0023] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, the workpiece surface image is an RGB three-channel color image.

[0024] Optionally, in the workpiece surface defect detection method based on the YOLACT model in any of the above embodiments, step S200 includes:

[0025] S210. Mark the surface defects in the acquired workpiece surface images and generate corresponding annotation files. One annotation file is generated for each workpiece surface image, including the type and location of the defect.

[0026] S220. After annotating all the collected workpiece surface images, obtain the annotation files corresponding to all workpiece surface images;

[0027] S230. Convert and merge the annotation files corresponding to all workpiece surface images into a single annotation file;

[0028] S240, the workpiece surface image and its corresponding annotation file constitute the workpiece surface defect dataset.

[0029] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, the annotation in step S210 uses the open-source annotation tool labelme.

[0030] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, the annotation file is in JSON format.

[0031] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, the location of the defect in step S210 is represented by a set of points of the defect boundary, forming a data structure of [[x1,y1],[x2,y2],...,[xn,yn]], where x1,y1,x2,y2,xn,yn represent the abscissa and ordinate of the first defect boundary point, the second defect boundary point, and the nth defect boundary point, respectively.

[0032] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, the format of the overall annotation file in step S230 is the same as the format of the MS COCO (Microsoft Common Objects in Context) dataset.

[0033] Optionally, in the workpiece surface defect detection method based on the YOLACT model in any of the above embodiments, the data augmentation strategy in step S300 includes:

[0034] Random cropping: Without removing defects from the workpiece surface, randomly crop the workpiece surface image and calculate the position of the original marked defect boundary points in the randomly cropped workpiece surface image.

[0035] Randomly flip the image horizontally, with a probability of 0.5, the workpiece surface image and the marked defect boundary points are flipped horizontally;

[0036] Random vertical flipping: The workpiece surface image and the marked defect boundary points are vertically flipped with a probability of 0.5.

[0037] Scale jitter: Randomly resize the workpiece surface image to 0.5-1.5 times its original size, and adjust the marked defect boundary points accordingly.

[0038] Color dithering converts the workpiece surface image from RGB space to HSV space by randomly changing the value, saturation, and hue of the workpiece surface image to form images under different lighting and colors. The transformed workpiece surface image is then converted back to RGB space.

[0039] Image stitching (Mosaic) involves randomly selecting four workpiece surface images from the workpiece surface defect dataset and then stitching these four workpiece surface images together. Each workpiece surface image has its corresponding labeled defect boundary points. After stitching the four workpiece surface images together, a new workpiece surface image is obtained, along with the corresponding labeled defect boundary points.

[0040] Image fusion (Mixup) involves randomly selecting two workpiece surface images from a workpiece surface defect dataset, and then weighting and fusing these two workpiece surface images according to a certain ratio, that is, adding the corresponding pixel values ​​according to a certain ratio.

[0041] Optionally, in the workpiece surface defect detection method based on the YOLACT model in any of the above embodiments, step S300 further includes: if the workpiece surface image is shifted or transformed after data augmentation, the position of the defect is changed accordingly by formula calculation to ensure that the defect in the workpiece surface image still matches the marked defect boundary.

[0042] Optionally, in the workpiece surface defect detection method based on the YOLACT model in any of the above embodiments, step S400 includes:

[0043] S410. Optimize the backbone network. The backbone network of the workpiece surface defect detection model adopts the ResNet18 network to extract multi-scale feature maps of the workpiece surface image.

[0044] S420. Add the GC-attention module to the ResNet18 network to improve the network's feature extraction capability.

[0045] S430, ordinary convolutional module replacement: Replace the ordinary convolutional modules in the ResNet18 network with depthwise separable convolutional modules, and use depthwise separable convolutional modules for convolution.

[0046] S440, Multi-scale Fusion: A BiFPN (Weighted Bidirectional Feature Pyramid Network) is used to fuse multi-scale feature maps extracted by the ResNet18 network, enhancing input features at different resolutions. The BiFPN network is a feature fusion network that adds contextual information edges to the original FPN (Feature Pyramid Networks) and multiplies each edge by a corresponding weight, enabling BiFPN to perform multi-scale feature map fusion more conveniently and quickly.

[0047] S450. The detection head of the YOLACT model is used as the detection head. The detection head includes a Protonet network, a Prediction Head module, and an NMS module. The Protonet network generates several template segmentation images. The Prediction Head module generates the category, bounding box, and combination coefficients of the template segmentation images for each target. The NMS module performs deduplication processing on the results generated by the Prediction Head module.

[0048] S460 improves edge detection performance by adding an edge detection branch, which runs in parallel with the BiFPN network and the detection head. The edge detection branch includes an edge feature extraction module and an edge detection head.

[0049] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, the input of the BiFPN network includes two paths: one is the multi-scale feature map output by ResNet18, and the other is the edge feature map output by the edge feature extraction module. The feature map output by the BiFPN network includes two paths: one is input to the Protonet network to predict the template segmentation image, and the other is input to the Prediction Head module and the NMS module to obtain the category, bounding box, and combination coefficient of each defect in the workpiece surface image and the template segmentation image.

[0050] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, during the training process of the workpiece surface defect detection model, a shallower feature layer of a ResNet18 network is selected as the input to the edge feature extraction module, and the output is an edge feature map. The edge feature map is divided into two paths. One path of the edge feature map is input to the BiFPN network for feature fusion, and the output is a feature map after feature fusion. The other path of the edge feature map is input to the edge detection head for training. The training label used by the edge detection head is obtained by extracting target edge information from the dataset label. When performing defect detection, the edge detection head is removed, and only the edge feature map is input to the BiFPN network for feature fusion.

[0051] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, step S420 includes:

[0052] S421. Add the GC-attention module to the ResNet18 network to model the contextual information of multi-scale feature maps;

[0053] S422. Feature recalibration is performed using modeling. The multi-scale feature map obtained from the ResNet18 network includes height dimension, width dimension and channel dimension.

[0054] S423. Perform feature recalibration on the multi-scale feature maps. The number of channels in the multi-scale feature maps is a multiple of 16, which increases exponentially with the number of network layers. Calculate the importance of each channel to all channels in the channel dimension for each multi-scale feature map, and use the importance of each channel to recalibrate the multi-scale feature maps to obtain higher quality feature maps.

[0055] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, step S430 includes:

[0056] S431. Perform convolution on each channel separately and then concatenate the outputs.

[0057] S432. Use a unit convolution kernel to perform convolution to obtain the feature map.

[0058] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, step S440 includes:

[0059] S441 and BiFPN networks remove nodes with only one input edge because nodes with only one input edge do not undergo feature fusion and contribute little to the fusion of multi-scale feature maps.

[0060] S442. If the original input node and the output node are in the same layer, add an extra edge between the original input node and the output node to fuse more features without increasing the additional computational cost.

[0061] The S443 and BiFPN networks treat each bidirectional path (i.e., top-down and bottom-up) as a feature network layer and repeat the same layer multiple times to achieve feature fusion at higher levels.

[0062] S444. Use fast normalization to fuse weighted features so that the values ​​of the normalized weights are between 0 and 1.

[0063] S445 and BiFPN networks integrate weighted bidirectional cross-scale connectivity and fast normalization fusion.

[0064] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, in step S450, the number of template segmentation images is 64.

[0065] Optionally, in the workpiece surface defect detection method based on the YOLACT model in any of the above embodiments, step S500 includes:

[0066] S510, the workpiece surface defect dataset is divided into training set and test set in a 4:1 ratio;

[0067] S520. The training set is augmented using the data augmentation strategy in step 300.

[0068] S530. Input the data-enhanced training set into the workpiece surface defect detection model for training;

[0069] S540. During the training process, the learning rate parameter is adjusted to make the loss function of the workpiece surface defect detection model converge, thus obtaining the initially trained workpiece surface defect detection model.

[0070] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, the total training loss of the workpiece surface defect detection model is a weighted sum of the classification loss and the regression loss: Loss = L xls +aL box +bL mask +cL edge ,

[0071] Among them, L cls For classification loss, focal loss is chosen as the loss between the defect category predicted by the model and the true defect category. Focal loss is a commonly used class loss function in object detection, addressing the problems of extreme imbalance between positive and negative samples and learning from difficult-to-classify samples in object detection; L box For the bounding box prediction loss, GIoU loss is chosen as the loss between the predicted defect bounding box coordinates and the ground truth bounding box coordinates; L mask For mask prediction loss, pixel-level binary cross-entropy loss is chosen as the loss between the predicted mask and the ground truth mask; L edge For edge prediction loss, pixel-level binary cross-entropy loss is selected as the loss between the predicted edge and the ground truth edge; a, b, and c are weight coefficients, preferably a, b, and c are all 1.

[0072] Furthermore, in the YOLACT model-based workpiece surface defect detection method in the above embodiments, the hyperparameter settings for training in step S530 are as follows: multi-scale training is adopted, and the size of the input workpiece surface image is adjusted so that the short side of the workpiece surface image is between 480 and 640 pixels, while the long side of the workpiece surface image does not exceed 1280 pixels; a stochastic gradient descent (SGD) optimizer with momentum of 0.9 and weight decay of 0.005 is adopted; the model is trained for a total of 100 epochs, with an initial learning rate of 0.001, and the learning rate is reduced to 1 / 10 of the original at the 70th and 90th epochs; training is carried out using 8 GPUs, with each GPU allocated two workpiece surface images, and the total batch size is 16.

[0073] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, step S600 includes:

[0074] S610. Input test set: Input the workpiece surface images in the test set into the preliminarily trained workpiece surface defect detection model.

[0075] S620, Multi-scale feature map extraction: Multi-scale feature maps are extracted using the ResNet18 network of the pre-trained workpiece surface defect detection model.

[0076] S630, Multi-scale fusion: The BiFPN network performs multi-scale fusion on the multi-scale feature maps extracted by the ResNet18 network;

[0077] S640: Output the predicted defect probability and predicted defect edge information. Use the detection head to process the multi-scale feature map output by the BiFPN network, and output the predicted defect probability and predicted defect edge information respectively.

[0078] S650: Display the type and location of defects. Based on the predicted defect probability and predicted defect edge information, display the type and location of defects in the workpiece surface image.

[0079] S660. Use the test set to verify the initially trained workpiece surface defect detection model. Use the mAP (mean Average Precision) index to calculate the average detection accuracy of all defect categories. When mAP is greater than the specified detection accuracy threshold, the workpiece surface defect detection model is verified; otherwise, return to step S500.

[0080] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, the detection accuracy threshold specified in step S660 is 0.6.

[0081] Furthermore, in the workpiece surface defect detection method based on the YOLACT model in the above embodiments, step S700 includes:

[0082] S710. Deploy validated workpiece surface defect detection models in industrial production lines;

[0083] S720, Run the workpiece surface defect detection model;

[0084] S730: Acquire images of workpiece surfaces from industrial production lines and use a workpiece surface defect detection model to perform real-time defect detection on the workpiece surface images.

[0085] S740: Output the workpiece surface defect detection results, displaying the type and location of the detected defects in the workpiece surface image.

[0086] This invention uses the YOLACT model, with a ResNet18 backbone network for multi-scale feature map extraction and a BiFPN network for deep feature fusion. The detection head of the YOLACT model is used as the detection head to obtain the final defect detection result. Simultaneously, an edge detection branch is added to the model to improve edge detection performance. Furthermore, this invention enhances the model's feature extraction capability and detection accuracy by adding a GC-attention module to the backbone network. This invention replaces ordinary convolutional modules in the backbone network with depthwise separable convolutional modules, achieving model lightweighting. Without sacrificing too much detection accuracy, it significantly improves detection speed and greatly reduces the computational cost of depthwise separable convolution, making it more suitable for deployment on CPUs and applicable to practical industrial production.

[0087] This invention has strong robustness and generalization ability, and improves the accuracy of defect target edge detection. Compared with the detection methods in the prior art, this invention can obtain more fine-grained detection results, higher detection speed, and is more suitable for deployment in CPUs, making it more suitable for application in actual industrial production environments.

[0088] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0089] Figure 1 This is a flowchart illustrating a workpiece surface defect detection method based on the YOLACT model according to an exemplary embodiment;

[0090] Figure 2 This is a schematic diagram illustrating the structure of a workpiece surface defect detection model based on the YOLACT model according to an exemplary embodiment. Detailed Implementation

[0091] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0092] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of components is schematically exaggerated in some places in the drawings.

[0093] The inventors designed a workpiece surface defect detection method based on the YOLACT model, such as... Figure 1 As shown, it includes the following steps:

[0094] S100, Image Acquisition: Using an industrial camera to acquire images of the workpiece surface on an industrial production line; specifically including:

[0095] S110, Illumination highlights the surface defects of workpieces on industrial production lines;

[0096] S120. Use an industrial camera to capture images of the workpiece surface. The workpiece surface image is an RGB three-channel color image.

[0097] S130, Store workpiece surface image.

[0098] S200, Image Annotation: Using image annotation tools, defects are annotated on the acquired workpiece surface images to generate a workpiece surface defect dataset; specifically including:

[0099] S210. Label the surface defects in the collected workpiece surface images. The labeling is done using the open-source labeling tool labelme to generate corresponding labeling files. One labeling file is generated for each workpiece surface image. The labeling file is in JSON format and includes the type and location of the defect. The location of the defect is represented by a set of points on the defect boundary, forming a data structure of [[x1,y1],[x2,y2],...,[xn,yn]]. Here, x1,y1,x2,y2,xn,yn represent the x-coordinate and y-coordinate of the 1st, 2nd, and nth defect boundary points, respectively.

[0100] S220. After annotating all the collected workpiece surface images, obtain the annotation files corresponding to all workpiece surface images;

[0101] S230. Convert and merge the annotation files corresponding to all workpiece surface images into a single annotation file. The format of the single annotation file is the same as that of the MS COCO (Microsoft Common Objects in Context) dataset.

[0102] S240, the workpiece surface image and its corresponding annotation file constitute the workpiece surface defect dataset.

[0103] S300. Data Augmentation: Apply data augmentation strategies to the workpiece surface defect dataset. If the workpiece surface image shifts or transforms after data augmentation, calculate the corresponding changes to the defect positions using formulas to ensure that the defects in the workpiece surface image still match the labeled defect boundaries. Data augmentation strategies include:

[0104] Random cropping: Without removing defects from the workpiece surface, randomly crop the workpiece surface image and calculate the position of the original marked defect boundary points in the randomly cropped workpiece surface image.

[0105] Randomly flip the image horizontally, with a probability of 0.5, the workpiece surface image and the marked defect boundary points are flipped horizontally;

[0106] Random vertical flipping: The workpiece surface image and the marked defect boundary points are vertically flipped with a probability of 0.5.

[0107] Scale jitter: Randomly resize the workpiece surface image to 0.5-1.5 times its original size, and adjust the marked defect boundary points accordingly.

[0108] Color dithering converts the workpiece surface image from RGB space to HSV space by randomly changing the value, saturation, and hue of the workpiece surface image to form images under different lighting and colors. The transformed workpiece surface image is then converted back to RGB space.

[0109] Image stitching (Mosaic) involves randomly selecting four workpiece surface images from the workpiece surface defect dataset and then stitching these four workpiece surface images together. Each workpiece surface image has its corresponding labeled defect boundary points. After stitching the four workpiece surface images together, a new workpiece surface image is obtained, along with the corresponding labeled defect boundary points.

[0110] Image fusion (Mixup) involves randomly selecting two workpiece surface images from a workpiece surface defect dataset, and then weighting and fusing these two workpiece surface images according to a certain ratio, that is, adding the corresponding pixel values ​​according to a certain ratio.

[0111] S400, Model Building: Constructing a workpiece surface defect detection model based on YOLACT; specifically including:

[0112] S410. Optimize the backbone network. The backbone network of the workpiece surface defect detection model adopts the ResNet18 network to extract multi-scale feature maps of the workpiece surface image.

[0113] S420. A GC-attention module is added to the ResNet18 network to improve its feature extraction capabilities; specifically including:

[0114] S421. Add the GC-attention module to the ResNet18 network to model the contextual information of multi-scale feature maps;

[0115] S422. Feature recalibration is performed using modeling. The multi-scale feature map obtained from the ResNet18 network includes height dimension, width dimension and channel dimension.

[0116] S423. Perform feature recalibration on the multi-scale feature maps. The number of channels in the multi-scale feature maps is a multiple of 16, which increases exponentially with the number of network layers. Calculate the importance of each channel to all channels in the channel dimension for each multi-scale feature map, and use the importance of each channel to recalibrate the multi-scale feature maps to obtain higher quality feature maps.

[0117] S430, ordinary convolutional module replacement: The ordinary convolutional modules in the ResNet18 network are replaced with depthwise separable convolutional modules, which are then used for convolution. Specifically, this includes:

[0118] S431. Perform convolution on each channel separately and then concatenate the outputs.

[0119] S432. Use a unit convolution kernel to perform convolution to obtain the feature map.

[0120] S440, Multi-scale Fusion: This method uses a BiFPN (Weighted Bidirectional Feature Pyramid Network) to fuse multi-scale feature maps extracted by the ResNet18 network, enhancing input features at different resolutions. The BiFPN is a feature fusion network that adds contextual information edges to the original FPN (Feature Pyramid Networks) and multiplies each edge by a corresponding weight, enabling more convenient and faster multi-scale feature map fusion. Specifically, it includes:

[0121] S441 and BiFPN networks remove nodes with only one input edge because they do not perform feature fusion and contribute little to the fusion of multi-scale feature maps.

[0122] S442. If the original input node and the output node are in the same layer, add an extra edge between the original input node and the output node to fuse more features without increasing the additional computational cost.

[0123] The S443 and BiFPN networks treat each bidirectional path (i.e., top-down and bottom-up) as a feature network layer and repeat the same layer multiple times to achieve feature fusion at higher levels.

[0124] S444. Use fast normalization to fuse weighted features so that the values ​​of the normalized weights are between 0 and 1.

[0125] S445 and BiFPN networks integrate weighted bidirectional cross-scale connectivity and fast normalization fusion.

[0126] S450. A detection head using the YOLACT model is adopted as the detection head. The detection head includes a Protonet network, a Prediction Head module, and an NMS module. The Protonet network generates several template segmentation images, preferably 64 template segmentation images. The Prediction Head module generates the category, bounding box, and combination coefficients of each target and template segmentation images. The NMS module performs deduplication processing on the results generated by the Prediction Head module. The input of the BiFPN network includes two paths: one is the multi-scale feature map output by ResNet18, and the other is the edge feature map output by the edge feature extraction module. The output feature map of the BiFPN network includes two paths: one is input to the Protonet network to predict the template segmentation image, and the other is input to the Prediction Head module and the NMS module to obtain the category, bounding box, and combination coefficients of each defect in the workpiece surface image.

[0127] S460. To improve edge detection performance, an edge detection branch is added, running in parallel with the BiFPN network and the detection head. This branch includes an edge feature extraction module and an edge detection head. During the training of the workpiece surface defect detection model, a shallower feature layer from a ResNet18 network is selected as input to the edge feature extraction module, which outputs an edge feature map. This edge feature map is divided into two paths: one path is input to the BiFPN network for feature fusion, outputting a fused feature map; the other path is input to the edge detection head for training. The training labels used by the edge detection head are obtained by extracting target edge information from the dataset labels. During defect detection, the edge detection head is removed, and only the edge feature map is input to the BiFPN network for feature fusion.

[0128] S500, Model Training: The data-enhanced workpiece surface defect dataset is divided into training and testing sets according to a certain ratio. The training set is input into the workpiece surface defect detection model for training. The learning rate parameters are adjusted to make the loss function of the workpiece surface defect detection model converge, resulting in a preliminarily trained workpiece surface defect detection model; specifically including:

[0129] S510, the workpiece surface defect dataset is divided into training set and test set in a 4:1 ratio;

[0130] S520. The training set is augmented using the data augmentation strategy in step 300.

[0131] S530. Input the data-enhanced training set into the workpiece surface defect detection model for training. The total training loss is the weighted sum of the classification loss and the regression loss:

[0132] Loss = L cls +aL box +bL mask +cL edge ,

[0133] Among them, L cls For classification loss, focal loss is chosen as the loss between the defect category predicted by the model and the true defect category. Focal loss is a commonly used class loss function in object detection, addressing the problems of extreme imbalance between positive and negative samples and learning from difficult-to-classify samples in object detection; L box For the bounding box prediction loss, GIoU loss is chosen as the loss between the predicted defect bounding box coordinates and the ground truth bounding box coordinates; L mask For mask prediction loss, pixel-level binary cross-entropy loss is chosen as the loss between the predicted mask and the ground truth mask; L edgeFor edge prediction loss, pixel-level binary cross-entropy loss is selected as the loss between the predicted edge and the ground truth edge; a, b, and c are weight coefficients, preferably all of 1; the hyperparameter settings for training are as follows: multi-scale training is adopted, and the size of the input workpiece surface image is adjusted so that the short side of the workpiece surface image is between 480-640 pixels, while the long side of the workpiece surface image does not exceed 1280 pixels; a stochastic gradient descent (SGD) optimizer with momentum of 0.9 and weight decay of 0.005 is adopted; the model is trained for a total of 100 epochs, with an initial learning rate of 0.001, and the learning rate is reduced to 1 / 10 of the original at the 70th and 90th epochs; training is carried out using 8 GPUs, with two workpiece surface images allocated to each GPU, and the total batch size is 16;

[0134] S540, adjust the learning rate parameter to make the loss function of the workpiece surface defect detection model converge, and obtain the initially trained workpiece surface defect detection model.

[0135] S600. Model Validation: Defect detection is performed using the pre-trained workpiece surface defect detection model, specifically by using the pre-trained model for defect prediction, resulting in a validated workpiece surface defect detection model. This includes:

[0136] S610. Input test set: Input the workpiece surface images in the test set into the preliminarily trained workpiece surface defect detection model.

[0137] S620, Multi-scale feature map extraction: Multi-scale feature maps are extracted using the ResNet18 network of the pre-trained workpiece surface defect detection model.

[0138] S630, Multi-scale fusion: The BiFPN network performs multi-scale fusion on the multi-scale feature maps extracted by the ResNet18 network;

[0139] S640: Output the predicted defect probability and predicted defect edge information. Use the detection head to process the multi-scale feature map output by the BiFPN network, and output the predicted defect probability and predicted defect edge information respectively.

[0140] S650: Display the type and location of defects. Based on the predicted defect probability and predicted defect edge information, display the type and location of defects in the workpiece surface image.

[0141] S660. Use the test set to verify the initially trained workpiece surface defect detection model. Use the mAP (mean Average Precision) index to calculate the average detection precision of all defect categories. When mAP is greater than the specified detection precision threshold, the validated workpiece surface defect detection model is obtained; otherwise, return to step S500. Preferably, the specified detection precision threshold is 0.6.

[0142] S700, Defect Detection: Deploying a validated workpiece surface defect detection model on industrial production lines for real-time workpiece surface defect detection; specifically including:

[0143] S710. Deploy validated workpiece surface defect detection models in industrial production lines;

[0144] S720, Run the workpiece surface defect detection model;

[0145] S730: Acquire images of workpiece surfaces from industrial production lines and use a workpiece surface defect detection model to perform real-time defect detection on the workpiece surface images.

[0146] S740 Output the workpiece surface defect detection results, displaying the type and location of the detected defects in the workpiece surface image.

[0147] The mAP (mean Average Precision) index was used to evaluate the detection performance of the workpiece surface defect detection model in the above embodiments. The mAP index reflects the detection performance of the workpiece surface defect detection model by calculating the average detection precision of all defect categories. The larger the mAP index, the better the detection performance. The workpiece surface defect detection method of the above embodiments was tested on a test set, and the calculated mAP index result was 0.653. Using existing surface defect detection models, such as Mask R-CNN and YOLACT, on the test set, the calculated mAP index results were 0.597 and 0.576, respectively. Therefore, it can be seen that the workpiece surface defect detection method of the above embodiments has a significant improvement in detection performance compared to existing surface defect detection models.

[0148] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for detecting surface defects in workpieces based on the YOLACT model, characterized in that, include: S100, Image Acquisition: Uses an industrial camera to acquire images of the workpiece surface on an industrial production line; S200, Image annotation: Use an image annotation tool to annotate defects on the workpiece surface image to generate a workpiece surface defect dataset; S300, Data Augmentation: Apply data augmentation strategies to the workpiece surface defect dataset to perform data augmentation. If the workpiece surface image shifts or changes after data augmentation, the position of the defect is adjusted accordingly by formula calculation to ensure that the defect in the workpiece surface image still matches the marked defect boundary. S400, Model Building: Construct a workpiece surface defect detection model based on YOLACT; include: S410. Optimize the backbone network. The backbone network of the workpiece surface defect detection model adopts the ResNet18 network to extract multi-scale feature maps of the workpiece surface image. S420. Add a GC-attention module. Add a GC-attention module to the ResNet18 network to improve the network's feature extraction capability. S430, Replacement of ordinary convolutional modules: Replace the ordinary convolutional modules in the ResNet18 network with depthwise separable convolutional modules, and use the depthwise separable convolutional modules for convolution. S440. Multi-scale fusion: The BiFPN network is used to perform multi-scale fusion on the multi-scale feature maps extracted by the ResNet18 network to enhance the input features at different resolutions. The input of the BiFPN network includes two paths: one is the multi-scale feature map output by ResNet18, and the other is the edge feature map output by the edge feature extraction module. The output feature map of the BiFPN network includes two paths: one is input to the Protonet network to predict the template segmentation image, and the other is input to the Prediction Head module and the NMS module to obtain the category of each defect in the workpiece surface image, the bounding box, and the combination coefficient of the template segmentation image. S450. The detection head using the YOLACT model is used as the detection head, which includes a Protonet network, a Prediction Head module, and an NMS module. S460. To improve edge detection performance, an edge detection branch is added, running in parallel with the BiFPN network and the detection head. The edge detection branch includes an edge feature extraction module and an edge detection head. S500, Model Training: The data-enhanced workpiece surface defect dataset is divided into a training set and a test set according to a certain ratio. The training set is input into the workpiece surface defect detection model for training to obtain a preliminarily trained workpiece surface defect detection model. S600, Model Validation: Defect detection is performed using the pre-trained workpiece surface defect detection model, i.e., defect prediction is performed using the pre-trained workpiece surface defect detection model to obtain a validated workpiece surface defect detection model. S700, Defect Detection, deploys a validated workpiece surface defect detection model on industrial production lines to perform real-time workpiece surface defect detection.

2. The workpiece surface defect detection method based on the YOLACT model as described in claim 1, characterized in that, Step S100 includes: S110, Illumination highlights the surface defects of the workpieces on the industrial production line; S120. Use the industrial camera to capture an image of the workpiece surface; S130. Store the image of the workpiece surface.

3. The workpiece surface defect detection method based on the YOLACT model as described in claim 2, characterized in that, Step S200 includes: S210. Mark the surface defects in the workpiece surface image and generate corresponding annotation files. One annotation file is generated for each workpiece surface image, including the type and location of the defect. S220. After completing the annotation of all workpiece surface images, the annotation files corresponding to all workpiece surface images are obtained. S230. Convert and merge the annotation files corresponding to all workpiece surface images into a single annotation file; S240. The workpiece surface image and its corresponding annotation file constitute a workpiece surface defect dataset.

4. The workpiece surface defect detection method based on the YOLACT model as described in claim 3, characterized in that, The data augmentation strategies in step S300 include: Random cropping: Without cropping out surface defects, the workpiece surface image is randomly cropped, and the positions of the original marked defect boundary points in the randomly cropped workpiece surface image are calculated. Randomly flip the image horizontally, with a probability of 0.5, the workpiece surface image and the marked defect boundary points are flipped horizontally; Randomly flip vertically, with a probability of 0.5, the workpiece surface image and the marked defect boundary points are vertically flipped; The scale is jittered, and the size of the workpiece surface image is randomly readjusted to 0.5-1.5 times the original size, and the marked defect boundary points are adjusted accordingly. Color dithering converts the workpiece surface image from RGB space to HSV space by randomly changing the brightness, saturation and hue of the workpiece surface image to form images under different lighting and colors, and then converts the transformed workpiece surface image back to RGB space. Image stitching: Four workpiece surface images are randomly selected from the workpiece surface defect dataset, and then the four workpiece surface images are stitched together to obtain a new workpiece surface image, and the corresponding marked defect boundary points are obtained at the same time. Image fusion involves randomly selecting two workpiece surface images from the workpiece surface defect dataset, and then weighting and fusing the two workpiece surface images according to a certain ratio, that is, adding the corresponding pixel values ​​according to a certain ratio.

5. The workpiece surface defect detection method based on the YOLACT model as described in claim 1, characterized in that, Step S420 includes: S421. Add a GC-attention module to the ResNet18 network to model the context information of the multi-scale feature map; S422. Feature recalibration is performed using modeling. The multi-scale feature map obtained by the ResNet18 network includes height dimension, width dimension and channel dimension. S423. Perform feature recalibration on the multi-scale feature map. The number of channels in the multi-scale feature map is a multiple of 16. For each multi-scale feature map, calculate the importance of each channel to all channels in the channel dimension. Use the importance of each channel to recalibrate the multi-scale feature map to obtain a higher quality feature map.

6. The workpiece surface defect detection method based on the YOLACT model as described in claim 5, characterized in that, Step S440 includes: S441. The BiFPN network deletes nodes with only one input edge; S442. If the original input node and the output node are in the same layer, add an extra edge between the original input node and the output node to fuse more features without increasing the additional computational cost. S443. The BiFPN network treats each bidirectional path, i.e., top-down and bottom-up, as a feature network layer and repeats the same layer multiple times to achieve feature fusion at higher levels. S444. Use fast normalization to fuse weighted features so that the values ​​of the normalized weights are between 0 and 1. S445. The BiFPN network integrates weighted bidirectional cross-scale connectivity and fast normalization fusion.

7. The workpiece surface defect detection method based on the YOLACT model as described in claim 6, characterized in that, Step S500 includes: S510. The workpiece surface defect dataset is divided into a training set and a test set in a 4:1 ratio. S520, The training set is augmented using the data augmentation strategy in step 300; S530. Input the data-enhanced training set into the workpiece surface defect detection model for training; S540. Adjust the learning rate parameter to make the loss function of the workpiece surface defect detection model converge, and obtain the initially trained workpiece surface defect detection model.

8. The workpiece surface defect detection method based on the YOLACT model as described in claim 7, characterized in that, Step S600 includes: S610. Input test set: Input the workpiece surface images in the test set into the preliminarily trained workpiece surface defect detection model. S620. Multi-scale feature map extraction: Extract multi-scale feature maps using the ResNet18 network of the pre-trained workpiece surface defect detection model. S630, Multi-scale fusion: The BiFPN network performs multi-scale fusion on the multi-scale feature maps extracted by the ResNet18 network; S640. Output the predicted defect probability and predicted defect edge information. Use the detection head to process the multi-scale feature map output by the BiFPN network, and output the predicted defect probability and predicted defect edge information respectively. S650, Display the type and location of the defect; based on the predicted defect probability and the predicted defect edge information, display the type and location of the defect in the workpiece surface image. S660. Use the test set to verify the initially trained workpiece surface defect detection model. Use the mAP index to calculate the average detection accuracy of all defect categories. When the mAP is greater than the specified detection accuracy threshold, a verified workpiece surface defect detection model is obtained; otherwise, return to step S500.

9. The workpiece surface defect detection method based on the YOLACT model as described in claim 8, characterized in that, Step S700 includes: S710. Deploy validated workpiece surface defect detection models in industrial production lines; S720. Run the workpiece surface defect detection model; S730. Acquire images of the workpiece surface of the industrial production line and use a workpiece surface defect detection model to perform real-time defect detection on the workpiece surface images. S740: Output the workpiece surface defect detection results, displaying the type and location of the detected defects in the workpiece surface image.

Images