Metal surface defect real-time detection method and system based on improved model

Abstract

The invention provides a metal surface defect real-time detection method and system based on an improved model, and the method comprises the steps: carrying out the real-time detection of surface defects of to-be-detected metal based on a pre-constructed improved model; wherein the step of pre-constructing the improved model comprises the following steps: carrying out first model improvement: replacing C2f modules of a preset part in a YOLOv8m model with a C2fASCF module and a C2fE-ASCF module; second model improvement is carried out, wherein an FFDPN module is added to the Neck part in the YOLOv8m model; third model improvement is carried out, wherein a detection head of the YOLOv8m model is replaced with an LE-SCDH detection head; and training the YOLOv8m model after the first model improvement, the second model improvement and the third model improvement by using the data set marked with the metal surface defects to obtain an improved model.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a real-time detection method and system for metal surface defects based on an improved model. Background Art

[0002] Strip steel has the characteristics of high strength, light weight, good plasticity, good corrosion resistance and recyclability. As a vital industrial raw material, it is increasingly widely used in many fields such as manufacturing machinery, automobiles, light industry, ships, aerospace, etc. However, in the actual industrial production process, factors such as poor quality of raw materials, harsh production environment, uneven production equipment, and insufficient processing technology often lead to a large number of defects on the surface of strip steel. The size of these defects ranges from hundreds of microns to tens of millimeters. If such defective products are not discovered in time, they will be broken, deformed and corroded during use after being supplied to other industries, which will reduce the service life of the product at the least and cause serious economic losses and casualties at the worst.

[0003] The development of surface defect detection technology is inseparable from the research of domestic and foreign scholars in recent decades. Current defect detection methods can be roughly divided into three categories: manual visual detection methods, traditional machine vision detection methods, and deep learning detection methods. Manual visual detection requires workers to work for a long time and at high intensity. Long-term high load will lead to increased missed detection rate and false detection rate, and a lot of training costs are required. Early machine vision-based methods require manual feature extraction, and then apply image processing technology and classification algorithms to classify defects. Texture features in images can reflect homogeneous patterns and organizational arrangements, and play a vital role in traditional machine vision methods. Although the above-mentioned traditional machine vision-based defect detection methods have basically realized the automated detection of surface defects, due to their carefully designed features, they are extremely sensitive to environmental factors such as light source intensity and different backgrounds, which makes it difficult to find the best artificial features to describe different defects, resulting in poor robustness and generalization of this type of defect detection method, and even requires "one scene, one method". Therefore, the possibility of its popularization in industrial production is very small. Deep learning methods make up for this shortcoming. The emergence of convolutional neural networks (CNNs) solves the problem of manual feature extraction. CNNs can automatically capture deep semantic features. CNN-based methods have stronger robustness and generalization capabilities than traditional methods. Convolutional neural networks have become a very important method in the industry. There are two types of target detection algorithms based on deep learning: two-stage and single-stage. The two-stage network is divided into two steps: generating proposed regions and image classification, and has high detection accuracy. Commonly used two-stage target detection algorithms include R-CNN, Fast R-CNN, Faster R-CNN, etc. The single-stage model directly performs classification and regression. Therefore, the detection speed of the single-stage algorithm is fast, but the accuracy is low, especially for overlapping targets and small targets. Commonly used single-stage target detection algorithms include SSD, YOLO, etc.

[0004] Defect detection has a very wide range of applications. However, metal surface defect detection tasks still have certain limitations. Directly using existing target detection models cannot well complete the task of detecting certain types of defects.

[0005] In fact, metal surface defect detection will face the following four main challenges:

[0006] 1) There are many types of defects and some defects have large intra-class differences. Some defect categories have prominent and dense shape features. At the same time, due to the influence of the production process and the surrounding environment, these defects show huge intra-class variations in shape and size.

[0007] 2) Some defect features are not obvious. The defect range of some defect categories covers a large area. At the same time, due to factors such as light and background color, their features are sparse and the boundaries are blurred, making it difficult to distinguish them from the background.

[0008] 3) A single image contains many defects of different categories, and there is a possibility of overlap between different defects, which increases the difficulty of the defect detection task to a certain extent.

[0009] 4) Balance between detection accuracy and detection speed. In actual industrial production, detection speed determines whether real-time detection can be achieved, while detection accuracy is related to the reliability and effectiveness of the results. It is necessary to improve the detection accuracy of the improved model as much as possible while meeting the requirements of real-time detection.

[0010] Therefore, a real-time detection method for metal surface defects based on an improved model is urgently needed to solve the above problems. Summary of the invention

[0011] One of the purposes of the present invention is to provide a real-time detection method for metal surface defects based on an improved model to solve the problems pointed out in the background technology.

[0012] An embodiment of the present invention provides a real-time detection method for metal surface defects based on an improved model, comprising:

[0013] Based on the pre-built improved model, the surface defects of the metal to be inspected can be detected in real time;

[0014] Among them, the pre-construction steps of the improved model include:

[0015] Improve the first model: replace the C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules;

[0016] Improve the second model: add FFDPN module to the Neck part of YOLOv8m model;

[0017] Improve the third model: replace the detection head of the YOLOv8m model with the LE-SCDH detection head;

[0018] Using a data set labeled with metal surface defects, the YOLOv8m model after the first model improvement, the second model improvement, and the third model improvement is trained to obtain an improved model.

[0019] Optionally, the C2f_ASCF module is a C2f module in which all Bottleneck modules are replaced with Bottleneck_ASCF modules; Bottleneck_ASCF includes an ASCF module; the ASCF module divides the input feature map into four groups, two of which use different convolution kernels for convolution operations, and the other two groups remain unchanged. After the convolution operation is completed, the four groups of feature maps are spliced ​​through the Concat module, and then the channel features are further fused through 1x1 convolution to achieve efficient integration of channel features, and finally generate an output feature map with rich context information.

[0020] Optionally, the C2f_E-ASCF module is a C2f module in which all Bottleneck modules are replaced by Bottleneck_E-ASCF modules; the E-ASCF module divides the input feature map into four groups, and applies 1x1, 3x3, 5x5 and 7x7 convolution operations to each group, merges the four groups of feature maps, and then fuses the channel features of each group through 1x1 convolution to generate the final output feature map.

[0021] Optionally, the FFDPN module consists of a FeatureFocus module and a feature diffusion mechanism; the FeatureFocus module receives inputs of three different scales, adopts an Inception-style design, and uses parallel deep convolutions to accurately select and aggregate rich information at multiple scales;

[0022] The FeatureFocus module aligns high-dimensional features and low-dimensional features with the features of the current layer through convolution and segmentation operations, and then forms a comprehensive feature map that integrates multi-level information through splicing operations; the comprehensive feature map is input into a set of parallel depth-wise separable convolutions for contextual feature extraction, and then the fused multi-scale features are further processed through 1×1 convolution to finally output the feature map.

[0023] Optionally, the LE-SCDH detection head uses a shared convolution strategy so that the output of the same convolutional layer can be used for multiple prediction tasks at the same time;

[0024] The LE-SCDH detection head uses 1×1 convolution and group normalization to reduce the dimension of the feature map, which not only reduces the computational load but also reduces the risk of overfitting while ensuring that the most critical feature information is retained;

[0025] The LE-SCDH detection head uses the first 3×3 shared convolutional layer to capture spatial information and focuses on semantic information through the second 3×3 shared convolutional layer;

[0026] The scale layer in the LE-SCDH detection head solves the inconsistency problem in multi-scale object detection by applying a scaling factor to each element in the channel.

[0027] Optionally, based on the pre-built improved model, real-time surface defect detection of the metal to be inspected is performed, including:

[0028] Acquire a surface image of the metal to be detected;

[0029] Based on the improved model and the surface image, the surface defects of the metal to be inspected are detected in real time.

[0030] Optionally, obtaining a surface image of the metal to be detected includes:

[0031] When the user enters the metal storage site, the drone is controlled to obtain the prediction basis of the user's metal usage intention; wherein the prediction basis of the metal usage intention includes at least: an image of the metal picked up by the user from the metal storage site and a work portrait of the user;

[0032] Based on the metal usage intention prediction basis, predict the user's first metal usage intention and second metal usage intention; wherein the first metal usage intention at least includes: metal size, metal type, metal shape; the second metal usage intention at least includes: acceptable metal surface defects;

[0033] Control the drone to search for multiple target metals that match the intended use of the first metal in the metal storage site and use them as metals to be detected;

[0034] Control the drone to shoot the surface of the metal to be inspected to obtain the surface image.

[0035] Optionally, after controlling the drone to photograph the surface of the metal to be inspected and obtaining the surface image, the method further includes:

[0036] When the detection result of real-time detection of surface defects of the metal to be detected based on the surface image based on the improved model is consistent with the intention of using the second metal, the drone is controlled to prompt the user to use the corresponding metal to be detected.

[0037] Optionally, the controlling the drone to photograph the surface of the metal to be detected includes:

[0038] Determine the distribution of the metal to be detected based on the stacking location of the metal to be detected in the metal stacking site;

[0039] Based on the metal cluster region condition, a metal cluster region is determined in the distribution of metals to be detected; wherein the metal cluster region condition at least includes: a straight-line distance between stacking positions of two metals to be detected in the same metal cluster region is less than a first distance threshold;

[0040] Control the drone to obtain the user's recent historical movement trajectory and historical clear visual range within the preset period;

[0041] Determine a target metal cluster area from the metal cluster area that meets a standard relative relationship with a historical moving trajectory and a historical clear visible range;

[0042] Control the drone to go to the target metal cluster area to take surface photos of the corresponding metal to be detected;

[0043] Among them, the standard relative relationship includes at least:

[0044] If the proportion of the first local trajectory of the historical movement trajectory falling into the first semicircle area exceeds the proportion of the second local trajectory of the historical movement trajectory falling into the second semicircle area, and the difference exceeds the proportion threshold, the overlapping area of ​​the metal cluster area and the second semicircle area exceeds the area threshold;

[0045] If the proportion difference does not exceed the proportion threshold, the metal cluster area does not overlap with the first semicircle area and the second semicircle area, and the closest distance between the metal cluster area and the first semicircle area or the second semicircle area does not exceed the second distance threshold; wherein a circular area is drawn with the target line segment connected at the beginning and the end of the historical moving trajectory as the diameter, and the target line segment divides the circular area into the first semicircle area and the second semicircle area;

[0046] The metal cluster area has not historically been completely within the historically clearly visible range;

[0047] The overlap of the metal cluster region that has historically partially entered the historically clear visible range does not exceed the overlap threshold.

[0048] An embodiment of the present invention provides a real-time metal surface defect detection system based on an improved model, comprising:

[0049] The detection module is used to detect surface defects of the metal to be inspected in real time based on the pre-built improved model;

[0050] Among them, the pre-construction steps of the improved model include:

[0051] Improve the first model: replace the C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules;

[0052] Improve the second model: add FFDPN module to the Neck part of YOLOv8m model;

[0053] Improve the third model: replace the detection head of the YOLOv8m model with the LE-SCDH detection head;

[0054] Using a data set labeled with metal surface defects, the YOLOv8m model after the first model improvement, the second model improvement, and the third model improvement is trained to obtain an improved model.

[0055] The present invention has achieved the following beneficial effects:

[0056] Using the improved YOLOv8m model, in order to improve the defect detection performance, we designed C2f_ASCF and C2f_E-ASCF modules to replace the C2f module of the original network to improve the feature extraction of multi-scale targets and reduce the FLOPS of the model; in the neck network, the designed FFDPN module uses channel perception to aggregate local and contextual information, and effectively propagates it to different scales through the diffusion mechanism, which significantly enhances the feature fusion ability of the model and enables it to fully utilize the complementarity between features of different scales; in the detection head part, we adopt the concept of weight sharing, further optimize and reconstruct the detection head, and design a lightweight and efficient shared convolutional detection head (LE-SCDH), which greatly reduces the number of model parameters and improves the detection performance of the model; the model with the highest accuracy is selected from the converged model as the final model, realizing high-speed, accurate and robust metal surface defect detection.

[0057] Optimize the extraction of defect features at different scales to improve the model's adaptability to defects with large changes in shape and size. This multi-scale feature extraction method can help the model better identify subtle differences, thereby improving the recognition accuracy of defects with large variations within the class.

[0058] The multi-channel information is effectively aggregated through the dimension-adaptive perception mechanism, thereby improving the recognition ability of sparse features of fuzzy boundaries. At the same time, by better integrating contextual information, the model can more accurately identify defect areas when dealing with light changes and background interference.

[0059] By improving the target detection algorithm to more accurately locate and classify overlapping defects, and by optimizing the production candidate areas, the model detection accuracy can be improved.

[0060] The internal modules of the target model are designed to be more lightweight to help the model ensure real-time detection while having higher detection accuracy.

[0061] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0062] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0064] Figure 1 The figure is an overall flow chart of a real-time metal surface defect detection method with adaptive scale context extraction and a lightweight and efficient mechanism according to an embodiment of the present invention.

[0065] Figure 2 The structure diagram of the innovative ASCF module and E-ASCF module according to the embodiment of the present invention;

[0066] Figure 3 The structure diagram of the C2f_ASCF and C2f_E-ASCF modules designed for the embodiments of the present invention;

[0067] Figure 4 LE-SCDH module structure diagram designed for an embodiment of the present invention;

[0068] Figure 5 The FFDPN module structure diagram designed for the embodiment of the present invention;

[0069] Figure 6 FeatureFocus module structure diagram designed for the embodiment of the present invention;

[0070] Figure 7 This is a structural diagram of the improved YOLOv8m model according to an embodiment of the present invention;

[0071] Figure 8 A data set image according to an embodiment of the present invention;

[0072] Fig. 9 It is a comparison chart of error type analysis radar of defect detection before and after the improvement of the model of the present invention;

[0073] Fig.10 This is a comparison chart of radar accuracy under COCO indicators (large, medium, and small) before and after the improvement of the model of the present invention;

[0074] Fig.11 This is a bar chart comparing the AP values ​​of the improved model of the present invention and the mainstream model on different defect categories;

[0075] Fig.12 The following is a comparison chart of the detection results of the baseline and improved models on this dataset. DETAILED DESCRIPTION

[0076] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0077] The embodiment of the present invention provides a real-time detection method for metal surface defects based on an improved model, such as Figure 1 As shown, including:

[0078] S1. Real-time detection of surface defects of metal to be inspected based on the pre-built improved model;

[0079] Among them, the pre-construction steps of the improved model include:

[0080] Improve the first model: replace the C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules;

[0081] Improve the second model: add FFDPN module to the Neck part of YOLOv8m model;

[0082] Improve the third model: replace the detection head of the YOLOv8m model with the LE-SCDH detection head;

[0083] Using a data set labeled with metal surface defects, the YOLOv8m model after the first model improvement, the second model improvement, and the third model improvement is trained to obtain an improved model.

[0084] The C2f_ASCF module is the C2f module in which all Bottleneck modules are replaced by Bottleneck_ASCF modules; Bottleneck_ASCF contains the ASCF module; the ASCF module divides the input feature map into four groups, two of which use different convolution kernels for convolution operations, and the other two groups remain unchanged. After the convolution operation is completed, the four groups of feature maps are spliced ​​through the Concat module, and then the channel features are further fused through 1x1 convolution to achieve efficient integration of channel features, and finally generate an output feature map with rich contextual information.

[0085] The C2f_E-ASCF module is the C2f module in which all Bottleneck modules are replaced by Bottleneck_E-ASCF modules; the E-ASCF module divides the input feature map into four groups, and applies 1x1, 3x3, 5x5 and 7x7 convolution operations to each group, merges the four groups of feature maps, and then fuses the channel features of each group through 1x1 convolution to generate the final output feature map.

[0086] The FFDPN module consists of a FeatureFocus module and a feature diffusion mechanism. The FeatureFocus module receives inputs of three different scales and adopts an Inception-style design, using parallel deep convolution to accurately select and aggregate rich information at multiple scales.

[0087] The FeatureFocus module aligns high-dimensional features and low-dimensional features with the features of the current layer through convolution and segmentation operations, and then forms a comprehensive feature map that integrates multi-level information through splicing operations; the comprehensive feature map is input into a set of parallel depth-wise separable convolutions for contextual feature extraction, and then the fused multi-scale features are further processed through 1×1 convolution to finally output the feature map.

[0088] The LE-SCDH detection head uses a shared convolution strategy so that the output of the same convolution layer can be used for multiple prediction tasks at the same time;

[0089] The LE-SCDH detection head uses 1×1 convolution and group normalization to reduce the dimension of the feature map, which not only reduces the computational load but also reduces the risk of overfitting while ensuring that the most critical feature information is retained;

[0090] The LE-SCDH detection head uses the first 3×3 shared convolutional layer to capture spatial information and focuses on semantic information through the second 3×3 shared convolutional layer;

[0091] The scale layer in the LE-SCDH detection head solves the inconsistency problem in multi-scale object detection by applying a scaling factor to each element in the channel.

[0092] The working principle and beneficial effects of the above technical solution are:

[0093] Traditional convolution usually only uses a single-scale convolution kernel, which limits the performance of the model when dealing with objects or scenes of different scales. This limitation is particularly prominent in metal surface defect detection due to the diversity and complexity of defects. To solve this problem, this patent proposes an adaptive scale context focus module (ASCF) and an extended adaptive scale context focus module (E-ASCF), which aim to better capture and fuse multi-scale features through multi-scale convolution kernels, thereby improving detection accuracy and efficiency.

[0094] refer to Figure 2As shown in the figure, the ASCF module divides the input feature map into four groups, two of which use different convolution kernels for convolution operations, while the other two groups remain unchanged. This design allows the model to capture defects such as small-scale micro cracks and large-scale scratches at the same time. After the convolution operation is completed, the four groups of feature maps are spliced ​​through the Concat module, and then the channel features are further fused through 1x1 convolution to achieve efficient integration of channel features, and finally generate output feature maps with rich contextual information. On this basis, the E-ASCF module further expands the capabilities of the ASCF module. The E-ASCF module also divides the input feature map into four groups, and applies 1x1, 3x3, 5x5 and 7x7 convolution operations to each group, merges the four groups of feature maps, and then fuses the channel features of each group through 1x1 convolution to generate the final output feature map. This design of applying convolution kernels of different scales in a larger range not only significantly improves the detection capability of complex defects, but also reduces the number of parameters and computational burden of the model, making it more suitable for real-time and large-scale industrial inspection scenarios. Figure 2 In the figure, C is the original number of channels, C / 2 is 1 / 2 of the original number of channels, C / 4 is 1 / 4 of the original number of channels, Split is a split operation, and Concat is a concatenation operation.

[0095] The original C2f module has insufficient feature extraction capability. Therefore, the C2f module of YOLOv8m is improved to the C2f_ASCF and C2f_E-ASCF modules innovated by the present invention, and most of the C2f modules in the entire model structure are replaced with C2f_ASCF and C2f_E-ASCF modules, which further enhances the ability of defect feature extraction and fusion in the network while ensuring a certain inference speed.

[0096] refer to Figure 3 As shown in the figure, the C2f_ASCF and C2f_E-ASCF structures replace all Bottleneck modules in the original C2f module with Bottleneck_ASCF and Bottleneck_E-ASCF modules. Based on the original Bottleneck module, the Bottleneck_ASCF and Bottleneck_E-ASCF modules further replace the traditional convolution with ASCF and E-ASCF modules. This operation not only optimizes the computational efficiency, but also reduces the number of parameters and computational burden of the model, and improves the overall performance of the defect detection model by accurately capturing the global structure and local details of metal surface defects. Figure 3 In the above example, + is a sum operation, Add is a sum operation, Split is a split operation, and Add? is a True operation or a False operation.

[0097] Metal surface defect detection is usually performed on edge devices with limited computing power, which places strict requirements on the lightweight and efficient detection model. At the same time, in order to solve the problem that the original YOLOv8m model performs poorly when dealing with small targets and complex backgrounds (such as occlusion and overlap), a lightweight and efficient shared convolutional detection head (LE-SCDH) is designed. The specific structure is as follows: Figure 4 As shown. Figure 4 In the equation, parameter sharing× is a parameter sharing operation, and parameter sharing√ is a parameter sharing cancellation operation.

[0098] LE-SCDH uses a shared convolution strategy, that is, sharing parameters between different tasks (such as bounding box regression and object classification), so that the output of the same convolution layer can be used for multiple prediction tasks at the same time. This not only effectively reduces the need for independent model learning parameters and redundant calculations, but also improves the adaptability and computational efficiency of the model. In order to capture and focus on multi-scale features, LE-SCDH uses 1×1 convolution and group normalization (GroupNorm) to reduce the dimension of the feature map, which not only reduces the computational load, but also reduces the risk of overfitting, while ensuring that the most critical feature information is retained. In the design of the shared convolution layer, the first 3×3 shared convolution layer is used to capture spatial information, while the second 3×3 shared convolution layer focuses on semantic information. This design ensures that the connection vector can contain enough spatial and semantic information to describe the location, size and category of the object. In order to further improve the detection performance, a scale layer is introduced to solve the inconsistency problem of the model in multi-scale object detection by applying a scaling factor to each element in the channel.

[0099] Through in-depth analysis of the parameter distribution of YOLOv8m, it was found that its detection head occupies 3.82 million parameters, accounting for about one-fifth of the parameters of the entire model. To this end, the detection head part was optimized and reconstructed, reducing the number of parameters from the original 3.82 million to 970,000, which laid a certain foundation for the deployment of the model in a resource-constrained environment.

[0100] FLOPS and Parameters are important indicators for measuring the computational complexity and resource requirements of deep learning models. FLOPS represents the total number of floating-point operations that the model needs to perform when processing data. The higher the FLOPS, the larger the model calculation, which means more computing power is needed to run; while Parameters represents the total number of parameters that need to be trained in the model. The more parameters and the larger the model, the higher the model storage requirements and memory usage during inference, and the greater the cost. From the experimental results in Table 1, we can see that the FLOPS and number of parameters of LE-SCDH only account for 78.76% and 25.39% of those of YOLOv8m-Head, and only account for 6.17% of the total number of model parameters, which significantly reduces the overall computational burden and achieves the lightweight detection head.

[0101] Table 1 Comparative analysis between the enhanced model and the YOLOv8m detection head

[0102]

[0103] like Fig.12 As shown in (a), according to the detection results of the unoptimized model, the detection effect of foreign matter (In) is the least ideal among all target types. The AP value is the lowest; the relative area of ​​this type of defect is different, and there are certain differences within the class; in order to improve the detection effect of the model for this type of situation, the FFDPN module is added to the last layer of the network Neck part to strengthen the processing and extraction of effective feature information from many defects of different scales;

[0104] refer to Figure 5 As shown in the figure, it is the structure diagram of the FFDPN module, which consists of the FeatureFocus module and the feature diffusion mechanism. The FeatureFocus module can receive inputs of three different scales, adopts the Inception-style design, and uses parallel deep convolution to accurately select and aggregate rich information at multiple scales. Figure 6As shown in the figure, first, the module aligns the high-dimensional features (P5) and low-dimensional features (P3) with the features of the current layer (P4) through convolution and segmentation operations. Subsequently, these processed feature maps are concatenated to form a feature map that integrates multi-level information. On this basis, the comprehensive feature map is input into a set of parallel depth-wise separable convolutions (5×5, 7×7, 9×9, 11×11) for contextual feature extraction. Next, the fused multi-scale features are further processed by 1×1 convolution. This operation not only realizes the effective fusion of local features and contextual features in the channel dimension, but also serves as a key channel fusion mechanism to effectively integrate features from different receptive fields. Ultimately, the feature map output by the module retains rich contextual information while avoiding excessive consumption of computation and memory. Figure 6 In the example, Identity is an identity mapping operation; Add is a sum operation; and Concat is a concatenation operation.

[0105] After completing the above three steps, you will get the improved and complete YOLOv8m model structure, refer to Figure 7 shown.

[0106] exist Fig. 9 Where Cls is a classification error, Loc is a positioning error, Both is a positioning error and a classification error, Dupe is a repeated detection error, Bkg is a Beijing error, FalsePos is a case where the prediction is true but the actual result is false; FalseNeg is a case where the prediction is false but the actual result is true.

[0107] The detection model is iteratively trained using the data set. After multiple trainings, the best model weights of the training are retained after the loss value stops decreasing. Through transfer learning, the best weights of each training are used for several iterations, and finally the detection model with the highest accuracy is selected as the metal surface defect detection model. When constructing the data set, the metal surface defect detection data set images are obtained; the images are resized to 640*640, and the collected data set images are randomly flipped, and data enhancement methods such as contrast adjustment, cropping, and scale transformation are used to expand the data set size; the labels in the data set are adapted, and corresponding labels are generated according to different image enhancement operations.

[0108] The following is the experimental process to prove that the metal surface defect detection capability of the improved model of this application is superior to other existing models:

[0109] The mainstream public dataset Deep Metal Surface Defect Dataset (GC10-DET) is used. The GC10-DET dataset contains 2257 images. The dataset also contains 10 defect categories: silk spots, inclusions, oil spots, indentations, water spots, creases, waist folds, punching holes, crescent bends and welds. The 10 defect categories are as follows Figure 5 As shown; there are 2250 images in total, and the image resolution is 2560*1960.

[0110] The Pytorch deep learning framework is used to train and test the proposed model. The experimental environment uses the following configuration: AMD EPYC7543CPU, RTXA5000GPU, 30GB memory, SGD optimizer is used to optimize the model, larger batch sizes are beneficial because they can improve the detection performance of the model, so this application uses a batch size of 16, 400 training rounds, and the image size is set to 640x 640. During the training process, random flipping, adjusting contrast, cropping, scale transformation and other data enhancement methods are used to improve the robustness of the model. In order to analyze whether the proposed improvements are effective, a combination experiment of each improvement strategy is carried out to control the variables.

[0111] The recall rate R (Recall), precision P (Precision), detection accuracy AP (AveragePrecision), average precision mAP (mean AveragePrecision) and the number of images processed per second FPS (Frames Per Second) are used as evaluation indicators; Fig.12 As shown, AP evaluates the accuracy of a given detection category by measuring the area under the Precision-Recall curve; mAP represents the average accuracy of all categories; frames per second (FPS) evaluates the real-time performance of the defect detection model by indicating the number of photos that can be analyzed per second. When calculating, the intersection over union (IoU) threshold is selected as 0.5, and IoU>0.5 is considered to be a successful detection of the target; the calculation formulas for these indicators are as follows:

[0112]

[0113]

[0114] TP, FP, and FN are the key metrics of the confusion matrix; TP represents the number of true positive samples correctly identified by the classifier, FP represents the number of false positive samples reported by the classifier, and FN represents the number of false negative samples. The i in mAP represents the specific category, and c represents the total number of detected categories. Finally, the parameters n and t in FPS represent the total number of images to be recognized and the time spent on detecting the image, respectively.

[0115] Among them, when all hyperparameters are the same, the experimental results of comparing the improved network model with other existing mainstream models are shown in Tables 1 and 2 below:

[0116] Table 2 Comparative experiments with mainstream models on the GC10-DET dataset

[0117]

[0118] Table 3 Comparison of AP50 values ​​of various defect categories on the GC10-DET dataset with mainstream models

[0119]

[0120] AP50 is the AP value mentioned above. Table 2 shows the experimental results comparison of the model with the other nine models on the GC10-DET validation dataset. As can be seen from the table, corresponding to mAP50, the improved network model has the highest score of 95.3%, which is 10.5% higher than the second-scoring model; for mAP50:95, the improved network model has a maximum score of 72.8%, and YOLOv10m ranks second at 54.0%. Compared with it, the improved network model is 18.8% higher. Compared with other excellent algorithms in the table, although it did not rank first in FPS, the improved network model can still achieve the highest detection accuracy while maintaining high real-time performance. At the same time, the above experimental results obtained on the GC10-DET dataset further show that the improved model of this application has good generalization performance and effectiveness.

[0121] Table 3 shows the AP50 experimental results of each defect category of the model compared with other models on the GC10-DET validation dataset. The table lists 11 rows of data, the first row represents the average AP50 value of all categories, and the remaining 10 rows represent the AP50 value of each category. Based on the data in Table 3, we also created Fig.11 to visualize the differences between the models. Fig.11 From each row of data in Table 3, we can see that the improved model has achieved the best detection effect in all defect categories. At the same time, the average detection accuracy of the improved model is much higher than that of other mainstream models in the table. These data further verify the superiority of the improved model in defect detection performance.

[0122] In one embodiment, based on the pre-built improved model, real-time detection of surface defects of the metal to be detected is performed, including:

[0123] Step S11, obtaining a surface image of the metal to be detected;

[0124] Step S12: Based on the improved model and the surface image, real-time detection of surface defects of the metal to be detected is performed.

[0125] The working principle and beneficial effects of the above technical solution are:

[0126] Based on the improved model, when the surface defects of the metal to be detected are detected in real time, the surface image of the metal to be detected is obtained, and the surface image is input into the improved model. The improved model will automatically perform surface defect analysis based on the surface image.

[0127] In one embodiment, the step of acquiring a surface image of the metal to be detected includes:

[0128] Step S111: When a user enters a metal storage site, the drone is controlled to obtain a prediction basis of the user's metal usage intention; wherein the prediction basis of the metal usage intention includes at least: an image of the metal picked up by the user from the metal storage site and a work portrait of the user;

[0129] In step S111, a large amount of metal with surface defects is piled up in the metal storage site; when the user enters the metal storage site, he hopes to find metal with surface defects within his acceptable range for use; the drone is stationed in the metal storage site. Therefore, when the user enters the metal storage site, the drone is controlled to obtain the prediction basis of the user's metal use intention; among them, the metal picked up by the user from the metal storage site refers to the metal that the user has picked out for use, and the acquisition of its image can control the drone to take pictures; the user's work portrait can be metal recycling tasks, preferred metal types, etc.

[0130] Step S112: based on the metal usage intention prediction basis, predict the user's first metal usage intention and second metal usage intention; wherein the first metal usage intention at least includes: metal size, metal type, and metal shape; the second metal usage intention at least includes: acceptable metal surface defects;

[0131] In step S112, the metal use intention prediction basis will continuously reflect which metal with surface defects the user wants, and therefore, the user's first metal use intention and second metal use intention can be predicted based on the metal use intention prediction basis;

[0132] Step S113, when the drone is controlled to search for multiple target metals that match the intended use of the first metal in the metal storage site, and use them as metals to be detected;

[0133] In step S113, when controlling the drone to search, the drone may be controlled to fly in the metal storage site to search;

[0134] Step S114: Control the drone to photograph the surface of the metal to be inspected to obtain a surface image.

[0135] In step S114, after the target metal is searched and used as the metal to be detected, the drone is controlled to take surface photos of the metal to be detected.

[0136] The embodiment of the present invention is to obtain a surface image in a special situation in which a user enters a metal storage site to search for metals with surface defects within an acceptable range for use.

[0137] In one embodiment, after controlling the drone to photograph the surface of the metal to be detected and obtaining the surface image, the method further includes:

[0138] Step S115: When the detection result of real-time surface defect detection of the metal to be detected based on the surface image based on the improved model is consistent with the second metal use intention, the drone is controlled to prompt the user to use the corresponding metal to be detected.

[0139] In step S115, after the surface image is obtained, the improved model will analyze the surface image to determine the surface defects to form the detection result; when the detection result is consistent with the intention of using the second metal, it means that the user will use the corresponding metal to be detected, and the drone will be controlled to prompt the user to use the corresponding metal to be detected. When prompting, an aperture can be projected onto the corresponding metal to be detected through the aperture projection device on the drone, so that the user can quickly discover it.

[0140] The embodiment of the present invention aims at the special situation that a user enters a metal storage site to search for metal with surface defects within an acceptable range for use. By controlling the drone, the user is assisted in quickly finding the metal within the acceptable range, thereby greatly improving the user's work efficiency.

[0141] In one embodiment, the controlling the drone to photograph the surface of the metal to be detected includes:

[0142] Step S1141, determining the distribution of the metal to be detected based on the stacking position of the metal to be detected in the metal stacking site;

[0143] In step S1141, after the drone searches for the metal to be detected, it also obtains its stacking location in the metal stacking site, and the stacking locations of multiple metals to be detected form a distribution of metals to be detected;

[0144] Step S1142: determining a metal cluster region in the distribution of metals to be detected based on a metal cluster region condition; wherein the metal cluster region condition at least includes: a straight-line distance between stacking positions of two metals to be detected in the same metal cluster region is less than a first distance threshold;

[0145] In step S1142, the metal cluster area refers to an area containing multiple stacking locations of the metal to be detected; the first distance threshold may be 3 meters; the straight-line distance between the stacking locations of two metals to be detected in the same metal cluster area is less than the first distance threshold, which can ensure that the drone can continuously take surface photos of all the metals to be detected in the same metal cluster area one by one;

[0146] Step S1143, controlling the drone to obtain the user's historical movement trajectory and historical clear visual range within a recent preset period;

[0147] In step S1143, the most recent preset time period refers to the time period within the last 5 minutes; the historical movement trajectory is the movement trajectory of the user in the metal storage yard within the most recent preset time period, which can be obtained based on the drone; the historical clear visual range is the clear visual range of the user in the metal storage yard within the most recent preset time period, which can be determined by the drone based on the user's eye direction, the normal clear visual distance of the naked eye, etc.;

[0148] Step S1144, determining a target metal cluster region from the metal cluster region that meets a standard relative relationship with a historical moving trajectory and a historical clear visible range;

[0149] In step S1144, when the target metal cluster area meets the standard relative relationship with the historical moving trajectory and the historical clear visible range, it means that controlling the drone to take surface photos of the metal to be detected in the target metal cluster area will not affect the user's current search work. If the target metal is found in the target metal cluster area, it is also convenient for the user to quickly pick the target metal.

[0150] Step S1145, controlling the drone to go to the target metal cluster area to take surface photos of the corresponding metal to be detected;

[0151] Among them, the standard relative relationship includes at least:

[0152] If the proportion of the first local trajectory of the historical movement trajectory falling into the first semicircle area exceeds the proportion of the second local trajectory of the historical movement trajectory falling into the second semicircle area, and the difference exceeds the proportion threshold, the overlapping area of ​​the metal cluster area and the second semicircle area exceeds the area threshold;

[0153] If the proportion difference does not exceed the proportion threshold, the metal cluster area does not overlap with the first semicircle area and the second semicircle area respectively, and the closest distance between the metal cluster area and the first semicircle area or the second semicircle area does not exceed the second distance threshold; wherein, a circular area is made with the target line segment connected at the beginning and the end of the historical moving trajectory as the diameter, and the target line segment divides the circular area into the first semicircle area and the second semicircle area; the closest distance is the smaller of the distance between the metal cluster area and the first semicircle area and the distance between the metal cluster area and the second semicircle area;

[0154] The metal cluster area has not historically been completely within the historically clearly visible range;

[0155] The overlap of the metal cluster region that has historically partially entered the historically clear visible range does not exceed the overlap threshold.

[0156] In step S1145, after the target metal cluster area is determined, the drone is controlled to go to the target metal cluster area to photograph the surface of the corresponding metal to be detected, ensuring the suitability of the drone to photograph the metal to be detected to obtain the surface image.

[0157] The historical movement trajectories are connected end to end to form a target segment. When a circular area is drawn with the target segment as the diameter, the midpoint of the target segment is the center of the circle and the half length of the target segment is the radius. The first local trajectory ratio is the ratio of the length of the local trajectory of the historical movement trajectory falling into the first semicircle area to the total length of the historical movement trajectory. The second local trajectory ratio is the ratio of the length of the local trajectory of the historical movement trajectory falling into the second semicircle area to the total length of the historical movement trajectory. The ratio difference refers to the difference between the first local trajectory ratio and the second local trajectory ratio. The larger ratio is the first local trajectory ratio, and the smaller ratio is the second local trajectory ratio. Trajectory proportion; the proportion threshold can be 3 / 10; the area threshold can be 1.5 square meters; the second distance threshold can be 2 meters; if the proportion of the first local trajectory in the first semicircle area of ​​the historical movement trajectory exceeds the proportion of the second local trajectory in the second semicircle area of ​​the historical movement trajectory, and the difference exceeds the proportion threshold, it means that the user's movement range in the recent preset time period involves both the first semicircle range and the second semicircle range, but involves the first semicircle range relatively more. At this time, it is ensured that the overlapping area of ​​the metal cluster area and the second semicircle area exceeds the area threshold. If the drone enters the metal cluster area, it will not affect the user's current search work. If The target metal is searched in the target metal cluster area, which is also convenient for the user to quickly pick the target metal; if the proportion difference does not exceed the proportion threshold, it means that the user's movement range in the recent preset time period involves both the first semicircle range and the second semicircle range, and the degree of involvement is similar. At this time, it is ensured that the metal cluster area does not overlap with the first semicircle area and the second semicircle area respectively, and the closest distance between the metal cluster area and the first semicircle area or the second semicircle area does not exceed the second distance threshold. If the drone enters the metal cluster area, it will not affect the user's current search work. If the target metal is searched in the target metal cluster area, it is also convenient for the user to quickly pick the target metal. picking, etc.; the overlap refers to the overlap between the areas where the metal cluster area has partially entered the historical clear visual range each time in history; the overlap threshold can be 20%; it is ensured that the metal cluster area has not completely entered the historical clear visual range in history, and the overlap of the areas where the metal cluster area has partially entered the historical clear visual range in history does not exceed the overlap threshold, then the metal cluster area can be constrained to be an area that the user has not fully and clearly visible but may want to search, then if the drone enters the metal cluster area, it will not affect the user's current search work, if the target metal is searched in the target metal cluster area, it is also convenient for the user to quickly pick the target metal, etc.

[0158] When the drone is taking surface photos of the metal to be detected, it may affect the user's current search work. If the target metal is found in the target metal cluster area, it may be inconvenient for the user to quickly pick the target metal. To address these problems, the embodiment of the present invention introduces a standard relative relationship, and determines from the metal cluster area a target metal cluster area that meets the standard relative relationship with the historical movement trajectory and the historical clear visible range, and controls the drone to go to the target metal cluster area to take surface photos of the corresponding metal to be detected. If the drone enters the metal cluster area, it will not affect the user's current search work. If the target metal is found in the target metal cluster area, it will also be convenient for the user to quickly pick the target metal, which greatly improves the rationality, applicability and accuracy of the system in acquiring surface images.

[0159] The embodiment of the present invention provides a real-time detection system for metal surface defects based on an improved model, comprising:

[0160] The detection module is used to detect surface defects of the metal to be inspected in real time based on the pre-built improved model;

[0161] Among them, the pre-construction steps of the improved model include:

[0162] Improve the first model: replace the C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules;

[0163] Improve the second model: add FFDPN module to the Neck part of YOLOv8m model;

[0164] Improve the third model: replace the detection head of the YOLOv8m model with the LE-SCDH detection head;

[0165] Using a data set labeled with metal surface defects, the YOLOv8m model after the first model improvement, the second model improvement, and the third model improvement is trained to obtain an improved model.

[0166] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A real-time detection method for metal surface defects based on an improved model, characterized in that: include: Based on the pre-built improved model, the surface defects of the metal to be inspected can be detected in real time; Among them, the pre-construction steps of the improved model include: Improve the first model: replace the C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules; Improve the second model: add FFDPN module to the Neck part of YOLOv8m model; Improve the third model: replace the detection head of the YOLOv8m model with the LE-SCDH detection head; Using a data set labeled with metal surface defects, the YOLOv8m model after the first model improvement, the second model improvement, and the third model improvement is trained to obtain an improved model.

2. The real-time detection method for metal surface defects based on the improved model according to claim 1, characterized in that: The C2f_ASCF module is the C2f module in which all Bottleneck modules are replaced by Bottleneck_ASCF modules; Bottleneck_ASCF contains the ASCF module; the ASCF module divides the input feature map into four groups, two of which use different convolution kernels for convolution operations, and the other two groups remain unchanged. After the convolution operation is completed, the four groups of feature maps are spliced ​​through the Concat module, and then the channel features are further fused through 1x1 convolution to achieve efficient integration of channel features, and finally generate an output feature map with rich contextual information.

3. The real-time detection method for metal surface defects based on the improved model according to claim 1, characterized in that: The C2f_E-ASCF module is the C2f module in which all Bottleneck modules are replaced by Bottleneck_E-ASCF modules; the E-ASCF module divides the input feature map into four groups, and applies 1x1, 3x3, 5x5 and 7x7 convolution operations to each group, merges the four groups of feature maps, and then fuses the channel features of each group through 1x1 convolution to generate the final output feature map.

4. The real-time detection method for metal surface defects based on the improved model according to claim 1, characterized in that: The FFDPN module consists of a FeatureFocus module and a feature diffusion mechanism. The FeatureFocus module receives inputs of three different scales and adopts an Inception-style design, using parallel deep convolution to accurately select and aggregate rich information at multiple scales. The FeatureFocus module aligns high-dimensional features and low-dimensional features with the features of the current layer through convolution and segmentation operations, and then forms a comprehensive feature map that integrates multi-level information through splicing operations; the comprehensive feature map is input into a set of parallel depth-wise separable convolutions for contextual feature extraction, and then the fused multi-scale features are further processed through 1×1 convolution to finally output the feature map.

5. The real-time detection method for metal surface defects based on the improved model according to claim 1, characterized in that: The LE-SCDH detection head uses a shared convolution strategy so that the output of the same convolution layer can be used for multiple prediction tasks at the same time; The LE-SCDH detection head uses 1×1 convolution and group normalization to reduce the dimension of the feature map, which not only reduces the computational load but also reduces the risk of overfitting while ensuring that the most critical feature information is retained; The LE-SCDH detection head uses the first 3×3 shared convolutional layer to capture spatial information and focuses on semantic information through the second 3×3 shared convolutional layer; The scale layer in the LE-SCDH detection head solves the inconsistency problem in multi-scale object detection by applying a scaling factor to each element in the channel.

6. The real-time detection method for metal surface defects based on the improved model according to claim 1, characterized in that: Based on the pre-built improved model, the surface defects of the metal to be inspected are detected in real time, including: Acquire a surface image of the metal to be detected; Based on the improved model and the surface image, the surface defects of the metal to be inspected are detected in real time.

7. The real-time detection method for metal surface defects based on the improved model according to claim 6, characterized in that: The step of obtaining a surface image of the metal to be detected comprises: When the user enters the metal storage site, the drone is controlled to obtain the prediction basis of the user's metal usage intention; wherein the prediction basis of the metal usage intention includes at least: an image of the metal picked up by the user from the metal storage site and a work portrait of the user; Based on the metal usage intention prediction basis, predict the user's first metal usage intention and second metal usage intention; wherein the first metal usage intention at least includes: metal size, metal type, metal shape; the second metal usage intention at least includes: acceptable metal surface defects; Control the drone to search for multiple target metals that match the intended use of the first metal in the metal storage site and use them as metals to be detected; Control the drone to shoot the surface of the metal to be inspected to obtain the surface image.

8. The real-time detection method for metal surface defects based on the improved model according to claim 7, characterized in that: Control the drone to take surface photos of the metal to be inspected. After obtaining the surface image, it also includes: When the detection result of real-time detection of surface defects of the metal to be detected based on the surface image based on the improved model is consistent with the intention of using the second metal, the drone is controlled to prompt the user to use the corresponding metal to be detected.

9. The real-time detection method for metal surface defects based on the improved model according to claim 7, characterized in that: The controlling drone to photograph the surface of the metal to be detected includes: Determine the distribution of the metal to be detected based on the stacking location of the metal to be detected in the metal stacking site; Based on the metal cluster region condition, a metal cluster region is determined in the distribution of metals to be detected; wherein the metal cluster region condition at least includes: a straight-line distance between stacking positions of two metals to be detected in the same metal cluster region is less than a first distance threshold; Control the drone to obtain the user's recent historical movement trajectory and historical clear visual range within the preset period; Determine a target metal cluster area from the metal cluster area that meets a standard relative relationship with a historical moving trajectory and a historical clear visible range; Control the drone to go to the target metal cluster area to take surface photos of the corresponding metal to be detected; Among them, the standard relative relationship includes at least: If the proportion of the first local trajectory of the historical movement trajectory falling into the first semicircle area exceeds the proportion of the second local trajectory of the historical movement trajectory falling into the second semicircle area, and the difference exceeds the proportion threshold, the overlapping area of ​​the metal cluster area and the second semicircle area exceeds the area threshold; If the proportion difference does not exceed the proportion threshold, the metal cluster area does not overlap with the first semicircle area and the second semicircle area, and the closest distance between the metal cluster area and the first semicircle area or the second semicircle area does not exceed the second distance threshold; wherein a circular area is drawn with the target line segment connected at the beginning and the end of the historical moving trajectory as the diameter, and the target line segment divides the circular area into the first semicircle area and the second semicircle area; The metal cluster area has not historically been completely within the historically clearly visible range; The overlap of the metal cluster region that has historically partially entered the historically clear visible range does not exceed the overlap threshold.

10. A real-time detection system for metal surface defects based on an improved model, characterized in that: include: The detection module is used to detect surface defects of the metal to be inspected in real time based on the pre-built improved model; Among them, the pre-construction steps of the improved model include: Improve the first model: replace the C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules; Improve the second model: add FFDPN module to the Neck part of YOLOv8m model; Improve the third model: replace the detection head of the YOLOv8m model with the LE-SCDH detection head; Using a data set labeled with metal surface defects, the YOLOv8m model after the first model improvement, the second model improvement, and the third model improvement is trained to obtain an improved model.

Images