A real-time detection method and system for metal surface defects based on an improved model

Abstract

The application provides a metal surface defect real-time detection method and system based on an improved model, wherein the method comprises: based on a pre-constructed improved model, performing real-time detection on the surface defects of a metal to be detected; wherein the pre-construction step of the improved model comprises: performing first model improvement: replacing the C2f module of a preset part in a YOLOv8m model with a C2f_ASCF module and a C2f_E-ASCF module; performing second model improvement: adding an FFDPN module to the Neck part in the YOLOv8m model; performing third model improvement: replacing the detection head of the YOLOv8m model with an LE-SCDH detection head; and training the YOLOv8m model after the first model improvement, the second model improvement and the third model improvement by using a data set labeled with metal surface defects, to obtain the improved model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer data processing, and particularly relates to a metal surface defect real-time detection method and system based on an improved model. BACKGROUND

[0002] The strip steel has the characteristics of high strength, light weight, good plasticity, good corrosion resistance and recyclability, and is used as an important industrial raw material, and is more and more widely used in many fields such as manufacturing machinery, automobiles, light industry, ships, aerospace and the like. However, in the actual industrial production process, factors such as low-quality raw materials, poor production environment, uneven production equipment, and insufficient processing technology often lead to a large number of defects on the surface of the strip steel. The size of these defects ranges from hundreds of microns to tens of millimeters. If such defective products are not discovered in time, they will cause rupture, deformation and corrosion in the use process after being supplied to other industries, which may reduce the service life of the products, or even cause serious economic losses and casualties.

[0003] The development of surface defect detection technology is inseparable from the research of scholars at home and abroad in recent decades. Current defect detection methods can be roughly divided into three categories: manual visual inspection method, traditional machine vision detection method and deep learning detection method. Manual visual inspection requires workers to work for a long time with high intensity. Long-term high load will lead to the increase of missed detection rate and false detection rate, and a large amount of training cost needs to be invested. Early methods based on machine vision need to manually set up feature extraction, and then apply image processing techniques and classification algorithms to classify defects. The texture features in the image can reflect the homogeneity pattern and organizational arrangement, which plays a crucial role in traditional machine vision methods. Although the above traditional machine vision defect detection method has basically realized the automatic detection of surface defects, due to the strong sensitivity of the carefully designed features to the intensity of the light source and different backgrounds and other environmental factors, it is difficult to find the best artificial features to describe different defects, which leads to poor robustness and generalization ability of this kind of defect detection method, and even needs “one scene one method”. Therefore, its possibility of popularization in industrial production is minimal. Deep learning methods make up for this shortcoming, and the emergence of convolutional neural network (CNN) solves the problem of manual feature extraction. CNN can automatically capture deep semantic features, and the method based on CNN has stronger robustness and generalization ability than traditional methods. Convolutional neural network has become a very important method in the industry. There are two types of target detection algorithms based on deep learning: two-stage and one-stage. The two-stage network is divided into two steps of generating suggestion area and image classification, and the detection accuracy is high. Common two-stage target detection algorithms include R-CNN, Fast R-CNN, Faster R-CNN, etc. The one-stage model directly classifies and regresses. Therefore, the detection speed of one-stage algorithm is fast, but the accuracy is low, especially for overlapping targets and small targets. Common one-stage target detection algorithms include SSD, YOLO, etc.

[0004] Defect detection has a very wide range of applications. However, there are still certain limitations in the task of metal surface defect detection. Direct use of existing target detection models cannot well complete the task of detecting some types of defects.

[0005] In fact, metal surface defect detection will face the following four main challenges:

[0006] 1) There are many types of defects and large intra-class differences in some defect classes. Some defect classes have prominent and dense shape features, and due to the influence of the production process and the surrounding environment, these defects show great intra-class variation in shape and size.

[0007] 2) The features of some defects are not obvious. The defect range of some defect classes covers a large area, and due to factors such as light and background color, the features are sparse and the boundaries are blurred, making it difficult to distinguish from the background.

[0008] 3) Single image contains a large number of different types of defects, and there is a possibility of overlapping different defects, which to some extent increases the difficulty of defect detection task.

[0009] 4) Balance between detection accuracy and detection speed. In actual industrial production, detection speed determines whether real-time detection can be realized, and detection accuracy relates to the reliability and effectiveness of the result. It is necessary to improve the detection accuracy of the model as much as possible under the premise of meeting real-time detection.

[0010] Therefore, there is an urgent need for a metal surface defect real-time detection method based on an improved model to solve the above problems. SUMMARY

[0011] One of the purposes of the present application is to provide a metal surface defect real-time detection method based on an improved model to solve the problems pointed out in the background art.

[0012] The metal surface defect real-time detection method based on an improved model provided by the embodiments of the present application comprises:

[0013] Based on the pre-constructed improved model, the surface defect of the metal to be detected is detected in real time;

[0014] The pre-construction step of the improved model comprises:

[0015] Performing first model improvement: replacing the C2f module of the preset part in the YOLOv8m model with the C2f_ASCF module and the C2f_E-ASCF module;

[0016] Performing second model improvement: adding an FFDPN module to the Neck part of the YOLOv8m model;

[0017] Performing third model improvement: replacing the detection head of the YOLOv8m model with an LE-SCDH detection head;

[0018] Using a data set labeled with metal surface defects, the YOLOv8m model after the first model improvement, the second model improvement and the third model improvement is trained to obtain an improved model.

[0019] Optionally, the C2f_ASCF module is a C2f module in which all Bottleneck modules are replaced by Bottleneck_ASCF modules; the Bottleneck_ASCF includes an ASCF module; the ASCF module divides the input feature map into four groups, two of which are subjected to convolution operation using different convolution kernels, and the other two groups remain unchanged; after the convolution operation, the four groups of feature maps are spliced through a Concat module, and then the channel features of each group are further fused through a 1x1 convolution to achieve efficient integration of the channel features, and finally an output feature map with rich context information is generated.

[0020] Optionally, the C2f_E-ASCF module is a C2f module in which all Bottleneck modules are replaced by Bottleneck_E-ASCF modules; the E-ASCF module divides the input feature map into four groups, and applies 1x1, 3x3, 5x5, and 7x7 convolution operations to each group respectively, combines the four groups of feature maps, and then fuses the channel features of each group through a 1x1 convolution to generate the final output feature map.

[0021] Optionally, the FFDPN module is composed of a FeatureFocus module and a feature diffusion mechanism; the FeatureFocus module receives three different scale inputs, adopts an Inception-style design, and uses parallel deep convolution to accurately select and aggregate rich information of multiple scales;

[0022] The FeatureFocus module aligns high-dimensional features and low-dimensional features with the features of the current layer through convolution and segmentation operations, and then splices the processed feature maps to form a comprehensive feature map that integrates multi-level information; the comprehensive feature map is input into a group of parallel deep separable convolutions for context feature extraction, and then the fused multi-scale features are further processed through a 1x1 convolution to output a final feature map.

[0023] Optionally, the LE-SCDH detection head uses a shared convolution strategy so that the output of the same convolution layer is used for multiple prediction tasks at the same time;

[0024] The LE-SCDH detection head uses 1x1 convolution and group normalization to reduce the dimensionality of the feature map, which not only reduces the computational load but also reduces the risk of overfitting while ensuring the preservation of the most critical feature information;

[0025] The LE-SCDH detection head uses a first 3x3 shared convolution layer to capture spatial information and a second 3x3 shared convolution layer to focus on semantic information;

[0026] The scale layer in the LE-SCDH detection head solves the inconsistency problem in multi-scale target detection by applying a scaling factor to each element in the channel.

[0027] Optionally, based on the pre-constructed improved model, the surface defects of the to-be-detected metal are detected in real time, comprising:

[0028] Obtaining a surface image of the to-be-detected metal;

[0029] Based on the improved model, the surface defects of the to-be-detected metal are detected in real time according to the surface image.

[0030] Optionally, the surface image of the to-be-detected metal is obtained, comprising:

[0031] When the user enters the metal stacking site, the metal use intention prediction basis of the user is obtained by controlling the UAV; wherein the metal use intention prediction basis at least includes: the image of the metal selected by the user from the metal stacking site, the work image of the user;

[0032] Based on the metal use intention prediction basis, the first metal use intention and the second metal use intention of the user are predicted; wherein the first metal use intention at least includes: metal size, metal type, metal shape; the second metal use intention at least includes: acceptable metal surface defects;

[0033] The UAV is controlled to search for a plurality of target metals in the metal stacking site that match the first metal use intention, and the target metals are used as to-be-detected metals;

[0034] The surface of the to-be-detected metal is photographed by controlling the UAV, and a surface image is obtained.

[0035] Optionally, after the surface of the to-be-detected metal is photographed by controlling the UAV to obtain the surface image, it further comprises:

[0036] When the detection result of the to-be-detected metal based on the improved model and the surface image is detected in real time is consistent with the second metal use intention, the user is prompted to use the corresponding to-be-detected metal by controlling the UAV.

[0037] Optionally, the surface of the to-be-detected metal is photographed by controlling the UAV, comprising:

[0038] Based on the stacking position of the to-be-detected metal in the metal stacking site, the distribution of the to-be-detected metal is determined;

[0039] Based on the metal cluster area condition, the metal cluster area is determined in the to-be-detected metal distribution; wherein the metal cluster area condition at least includes: the straight line distance between the stacking positions of two to-be-detected metals in the same metal cluster area is less than a first distance threshold;

[0040] The unmanned aerial vehicle is controlled to obtain a historical movement track and a historical clear visual range in a preset time period of a user in the past;

[0041] The target metal cluster area that meets a standard relative relationship with the historical movement track and the historical clear visual range is determined from the metal cluster area;

[0042] The unmanned aerial vehicle is controlled to go to the target metal cluster area to take a surface photograph of the corresponding metal to be detected;

[0043] The standard relative relationship at least includes:

[0044] If the proportion difference between the first local track proportion of the historical movement track falling into the first semicircular area and the second local track proportion of the historical movement track falling into the second semicircular area exceeds a proportion threshold, and the overlapping area of the metal cluster area and the second semicircular area exceeds an area threshold;

[0045] If the proportion difference does not exceed the proportion threshold, the metal cluster area does not overlap with the first semicircular area and the second semicircular area, respectively, and the closest distance between the metal cluster area and the first semicircular area or the second semicircular area does not exceed a second distance threshold; wherein a target line segment with the historical movement track connected at both ends is taken as the diameter of a circular area, and the target line segment divides the circular area into a first semicircular area and a second semicircular area;

[0046] The metal cluster area has never completely entered the historical clear visual range in the past;

[0047] The overlapping degree of the metal cluster area that has partially entered the area within the historical clear visual range in the past does not exceed an overlapping degree threshold.

[0048] An embodiment of the present application provides a metal surface defect real-time detection system based on an improved model, which comprises:

[0049] A detection module is configured to perform real-time detection of surface defects of the metal to be detected based on the pre-constructed improved model;

[0050] The pre-construction step of the improved model comprises:

[0051] First model improvement: replace the C2f module in the preset part of the YOLOv8m model with the C2f_ASCF module and the C2f_E-ASCF module;

[0052] Second model improvement: add the FFDPN module to the Neck part of the YOLOv8m model;

[0053] Third model improvement: replace the detection head of the YOLOv8m model with the LE-SCDH detection head;

[0054] The improved YOLOv8m model is trained by using a data set labeled with metal surface defects to obtain an improved model.

[0055] The present application has the following advantages:

[0056] The improved YOLOv8m model is used to improve the defect detection performance, and C2f_ASCF and C2f_E-ASCF modules are designed to replace the original network C2f module to improve the feature extraction of multi-scale targets and reduce the FLOPS of the model; in the neck network, the designed FFDPN module uses channel perception to aggregate local and contextual information, and effectively propagates it to different scales through a diffusion mechanism, significantly enhancing the feature fusion capability of the model, so that it can fully utilize the complementarity between different scale features; in the detection head part, the weight sharing concept is adopted, and the detection head is further optimized and reconstructed, and a lightweight and efficient shared convolution detection head (LE-SCDH) is designed, which greatly reduces the number of model parameters and improves the detection performance of the model; the model with the highest accuracy is selected from the converged model as the final model, realizing high-speed, accurate and robust metal surface defect detection.

[0057] The defect features of different scales are optimized and extracted to improve the adaptability of the model to defects with large shape and size changes. This multi-scale feature extraction method can help the model better recognize subtle differences, thereby improving the recognition accuracy of defects with large intra-class changes.

[0058] The multi-channel information is effectively aggregated through the dimension self-adaptive perception mechanism, thereby improving the recognition ability of the sparse features of the fuzzy boundary. At the same time, by better integrating the context information, the model can more accurately identify the defect area when dealing with light changes and background interference.

[0059] The improved target detection algorithm is used to more accurately locate and classify overlapping defects, and the model detection accuracy is improved by optimizing the production candidate area.

[0060] The internal modules of the target model are designed to be more lightweight to help the model ensure real-time detection while having higher detection accuracy.

[0061] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the written description and drawings.

[0062] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0063] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and are meant to explain the application and are not intended to limit the application. In the drawings:

[0064] Figure 1 The overall flow chart of the real-time metal surface defect detection method with adaptive scale context extraction and light and efficient mechanism of the embodiments of the application.

[0065] Figure 2 The structure diagram of the innovative ASCF module and E-ASCF module of the embodiments of the application;

[0066] Figure 3 The structure diagram of the C2f_ASCF and C2f_E-ASCF modules designed for the embodiments of the application;

[0067] Figure 4 The structure diagram of the LE-SCDH module designed for the embodiments of the application;

[0068] Figure 5 The structure diagram of the FFDPN module designed for the embodiments of the application;

[0069] Figure 6 The structure diagram of the FeatureFocus module designed for the embodiments of the application;

[0070] Figure 7 The structure diagram of the improved YOLOv8m model of the embodiments of the application;

[0071] Figure 8 The dataset image of the embodiments of the application;

[0072] Figure 9 The error type analysis radar comparison chart of the defect detection before and after the model improvement of the application;

[0073] Figure 10 The radar precision comparison chart under the COCO index (three levels of large, medium and small) before and after the model improvement of the application;

[0074] Figure 11 The AP value bar comparison chart of the improved model and the mainstream model on different defect categories of the application;

[0075] Figure 12 The detection result comparison chart of the baseline and the improved model on the dataset. DETAILED DESCRIPTION

[0076] The preferred embodiments of the present application are described below with reference to the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0077] The embodiment of the present application provides a metal surface defect real-time detection method based on an improved model, as shown in the figure, comprising: Figure 1

[0078] S1, based on a pre-constructed improved model, performing surface defect real-time detection on a metal to be detected;

[0079] The pre-construction step of the improved model comprises:

[0080] Performing first model improvement: replacing all C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules;

[0081] Performing second model improvement: adding an FFDPN module to the Neck part in the YOLOv8m model;

[0082] Performing third model improvement: replacing the detection head of the YOLOv8m model with an LE-SCDH detection head;

[0083] Using a data set labeled with metal surface defects, training the YOLOv8m model after the first model improvement, the second model improvement and the third model improvement to obtain the improved model.

[0084] The C2f_ASCF module is a C2f module in which all Bottleneck modules are replaced with Bottleneck_ASCF modules; the Bottleneck_ASCF includes an ASCF module; the ASCF module divides the input feature map into four groups, two of which are subjected to convolution operation using different convolution kernels, and the other two groups remain unchanged; after the convolution operation, the four groups of feature maps are spliced through a Concat module, and then the channel features of each group are further fused through a 1x1 convolution to realize efficient integration of the channel features, and finally an output feature map with rich context information is generated.

[0085] The C2f_E-ASCF module is a C2f module in which all Bottleneck modules are replaced with Bottleneck_E-ASCF modules; the E-ASCF module divides the input feature map into four groups, and applies 1x1, 3x3, 5x5 and 7x7 convolution operations to each group respectively, combines the four groups of feature maps, and then fuses the channel features of each group through a 1x1 convolution to generate the final output feature map.

[0086] ​The FFDPN module is composed of a FeatureFocus module and a feature diffusion mechanism; the FeatureFocus module receives inputs of three different scales, adopts an Inception-style design, and uses parallel deep convolution to accurately select and aggregate rich information of multiple scales;

[0087] The FeatureFocus module aligns high-dimensional features and low-dimensional features with the features of the current layer through convolution and segmentation operations, and then forms a comprehensive feature map that integrates multi-level information through a splicing operation; the comprehensive feature map is input into a group of parallel depth separable convolutions for context feature extraction, and then the fused multi-scale features are further processed by 1x1 convolution to output the final feature map.

[0088] The LE-SCDH detection head uses a shared convolution strategy, so that the output of the same convolution layer is used for multiple prediction tasks at the same time;

[0089] The LE-SCDH detection head uses 1x1 convolution and group normalization to reduce the dimension of the feature map, which not only reduces the computational load, but also reduces the risk of overfitting, while ensuring the preservation of the most critical feature information;

[0090] The LE-SCDH detection head uses a first 3x3 shared convolution layer to capture spatial information, and a second 3x3 shared convolution layer to focus on semantic information;

[0091] The scale layer in the LE-SCDH detection head solves the inconsistency problem in multi-scale target detection by applying a scaling factor to each element in the channel.

[0092] The working principle and beneficial effects of the above technical solutions are:

[0093] Traditional convolution usually only uses a single scale convolution kernel, which limits the performance of the model in processing objects or scenes of different scales, especially in metal surface defect detection, due to the diversity and complexity of defects. To solve this problem, the patent proposes an adaptive scale context focusing module (ASCF) and an extended adaptive scale context focusing module (E-ASCF), which aims to better capture and fuse multi-scale features through multi-scale convolution kernels, thereby improving the accuracy and efficiency of detection.

[0094] Reference Figure 2As shown, the ASCF module divides the input feature map into four groups, two of which are convolved using different convolution kernels, while the other two remain unchanged. This design allows the model to capture both small-scale microcracks and large-scale scratches and other defects. After the convolution operation, the four groups of feature maps are spliced through the Concat module, and then further fused through 1x1 convolution to achieve efficient integration of channel features, and finally generate output feature maps with rich contextual information. On this basis, the E-ASCF module further expands the capabilities of the ASCF module. The E-ASCF module also divides the input feature map into four groups, and applies 1x1, 3x3, 5x5, and 7x7 convolution operations to each group, respectively. The four groups of feature maps are merged, and then the channel features of each group are fused through 1x1 convolution to generate the final output feature map. This design of applying different size convolution kernels in a larger range not only significantly improves the detection capability of complex defects, but also reduces the parameter amount and computational burden of the model, making it more suitable for real-time and large-scale industrial detection scenarios. In Figure 2 In the formula, C is the original channel number, C / 2 is the 1 / 2 original channel number, C / 4 is the 1 / 4 original channel number, Split is the split operation, and Concat is the splicing operation.

[0095] The original C2f module has insufficient feature extraction capability. Therefore, the C2f module of YOLOv8m is improved to the C2f_ASCF and C2f_E-ASCF modules of the present application, and most of the C2f modules in the entire model structure are replaced with the C2f_ASCF and C2f_E-ASCF modules, which further enhance the defect feature extraction and fusion capability of the network under the premise of ensuring a certain inference speed.

[0096] Reference Figure 3 As shown, the C2f_ASCF and C2f_E-ASCF structures replace all Bottleneck modules in the original C2f module with Bottleneck_ASCF modules and Bottleneck_E-ASCF modules. The Bottleneck_ASCF module and the Bottleneck_E-ASCF module further replace the traditional convolution with the ASCF and E-ASCF modules based on the original Bottleneck module. This operation not only optimizes the calculation efficiency, but also reduces the parameter amount and computational burden of the model, and improves the overall performance of the defect detection model by accurately capturing the global structure and local details of the metal surface defects. Figure 3 In the formula, + is the summation operation, Add is the summation operation, Split is the split operation, and Add? is the True operation or False operation.

[0097] Metal surface defect detection is usually performed on edge devices with limited computing power, so there is a strict requirement for lightweight and efficient detection models. At the same time, in order to solve the problem that the original YOLOv8m model performs poorly when dealing with small targets and complex backgrounds (such as occlusion and overlap), a lightweight and efficient shared convolution detection head (LE-SCDH) is designed, and the specific structure is as shown in Figure 4 Figure 4 In Figure 4 , parameter sharingx is a parameter sharing operation, and parameter sharing is a parameter sharing operation.

[0098] LE-SCDH uses a shared convolution strategy, that is, sharing parameters between different tasks (such as bounding box regression and target classification), so that the output of the same convolution layer can be used for multiple prediction tasks at the same time. This not only effectively reduces the need for independent learning parameters of the model and reduces redundant calculations, but also improves the adaptability and computational efficiency of the model. In order to capture and focus on multi-scale features, LE-SCDH uses 1x1 convolution and group normalization (GroupNorm) to reduce the dimensionality of the feature map, which not only reduces the computational load, but also reduces the risk of overfitting, while ensuring that the most critical feature information is retained. In the design of the shared convolution layer, the first 3x3 shared convolution layer is used to capture spatial information, while the second 3x3 shared convolution layer focuses on semantic information. This design ensures that the connection vector can contain enough spatial and semantic information to describe the position, size and category of the object. In order to further improve the detection performance, a scale layer is also introduced, which applies a scaling factor to each element in the channel, thereby solving the inconsistency problem of the model when detecting multiple scale targets.

[0099] Through in-depth analysis of the parameter distribution of YOLOv8m, it is found that the detection head accounts for 3.82 million parameters, about one fifth of the entire model parameters. Therefore, the detection head part is optimized and reconstructed, reducing the parameter amount from the original 3.82 million to 970,000, which lays a certain foundation for deploying the model in resource-constrained environments.

[0100] FLOPS and Parameters are important indicators to measure the computational complexity and resource requirements of deep learning models. FLOPS represents the total number of floating-point operations required by the model when processing data. The higher the FLOPS, the greater the model's computation, meaning more computing power is needed to run it. Parameters, on the other hand, represent the total number of parameters that need to be trained in the model. The more parameters there are, the larger the model, and the higher the storage requirements and memory usage during inference, resulting in higher costs. From the experimental results in Table 1, it can be seen that the FLOPS and parameter quantity of LE-SCDH are only 78.76% and 25.39% of YOLOv8m-Head, and only account for 6.17% of the total model parameter quantity, significantly reducing the overall computational burden and achieving lightweight detection heads.

[0101] Table 1 Comparison between enhanced models and YOLOv8m detection heads

[0102]

[0103] As shown in Figure 12 (a), according to the detection results of the unoptimized model, the detection effect of foreign matter (In) in all target categories is the least ideal. The AP value is the lowest; the relative area of this defect is not uniform, and there is also a certain difference within the class. In order to improve the detection effect of the model on this kind of situation, the FFDPN module is added to the last layer of the network Neck part to strengthen the processing and extraction of effective feature information from a large number of different scale defects.

[0104] Referring to Figure 5 , it is a structural diagram of the FFDPN module, which consists of a FeatureFocus module and a feature diffusion mechanism. The FeatureFocus module can receive three different scale inputs, uses an Inception-style design, and utilizes parallel deep convolution to accurately select and aggregate rich information of multiple scales. The structural diagram of the FeatureFocus module is as shown in Figure 6As shown, first, the module aligns the high-dimensional features (P5) and low-dimensional features (P3) with the features of the current layer (P4) through convolution and segmentation operations, and then forms a feature map that integrates multi-level information through a splicing operation; on this basis, the integrated feature map is input into a group of parallel depth separable convolutions (5x5, 7x7, 9x9, 11x11) for context feature extraction, followed by further processing of the fused multi-scale features through 1x1 convolution. This operation not only realizes the effective fusion of local features and context features in the channel dimension, but also serves as a key channel fusion mechanism to effectively integrate features from different receptive fields. Finally, the feature map output by the module retains rich context information while avoiding excessive consumption of computation and memory. Figure 6 In the above three steps, Identity is the identity mapping operation; Add is the summation operation; and Concat is the splicing operation.

[0105] After completing the work of the above three steps, the improved complete YOLOv8m model structure is obtained, as shown in Figure 7 .

[0106] In the above three steps, Identity is the identity mapping operation; Add is the summation operation; and Concat is the splicing operation. Figure 9 In the above three steps, Identity is the identity mapping operation; Add is the summation operation; and Concat is the splicing operation.

[0107] The detection model is iteratively trained using the dataset. Through multiple training, the best model weight of this training is retained after the loss value stops declining. Through transfer learning, the best weight of each training is used to perform several iterations, and finally the detection model with the highest precision is selected as the metal surface defect detection model. In the process of constructing the dataset, the metal surface defect detection dataset images are obtained; the images are resized to 640*640, and the collected dataset images are subjected to random flipping, contrast adjustment, cropping, scale transformation and other data enhancement methods to expand the dataset size; the labels in the dataset are adapted, and the corresponding labels are generated according to different image enhancement operations.

[0108] The following is the proof experiment process that the improved model of the present application has better metal surface defect detection capability than other existing models:

[0109] The mainstream public dataset deep metal surface defect dataset (GC10-DET) is adopted, the GC10-DET dataset contains 2257 images, the dataset also contains 10 defect categories of silk spots, inclusions, oil spots, indentations, water spots, creases, waist folds, punching, crescent bends and welds, 10 defect categories are as shown in Figure 5

[0110] The Pytorch deep learning framework is used to train and test the proposed model. The experimental environment adopts the following configuration: AMD EPYC7543 CPU, RTXA5000 GPU, 30GB memory, SGD optimizer is used to optimize the model, and larger batch size is beneficial because they can improve the detection performance of the model, therefore, the batch size is 16, the training round is 400, and the picture size is all set to 640x 640. In the training process, random flip, contrast adjustment, cropping, scaling transformation and other data enhancement methods are used to improve the robustness of the model. In order to analyze whether the improvements of each part are effective, combination experiments of each improvement strategy are carried out to control the variables.

[0111] Recall R (Recall), precision P (Precision), detection precision AP (Average Precision), average precision value mAP (mean Average Precision) and frames per second FPS (Frames Per Second) are used as evaluation indexes; as shown in Figure 12 AP, the precision of a given detection category is evaluated by measuring the area under the Precision-Recall curve; mAP represents the average precision of all categories; the frame rate per second (FPS) evaluates the real-time performance of the defect detection model by indicating the number of photos that can be analyzed per second. When calculating, the intersection over union IoU threshold is selected as 0.5, and IoU>0.5 is considered as successfully detecting the target; the calculation formula of these indexes is as follows:

[0112]

[0113]

[0114] Where TP, FP and FN are the key metrics of the confusion matrix; TP represents the number of true positive samples correctly identified by the classifier, FP represents the number of false positive samples reported by the classifier, and FN represents the number of false negative samples. In mAP, i represents a specific category, and c represents the total number of detection categories. Finally, in FPS, parameters n and t represent the total number of identified images and the time spent on detecting the image respectively.

[0115] ​Wherein, under the condition that each hyperparameter is the same, the improved network model is compared with other existing mainstream models, and the comparison experimental results are shown in Table 1 and Table 2.

[0116] Table 2 Comparison of experimental results of GC10-DET data set with mainstream models

[0117]

[0118] Table 3 Comparison of AP50 values of each defect category of GC10-DET data set with mainstream models

[0119]

[0120] AP50 is the above-mentioned AP value. Table 2 shows the comparison of experimental results of the model with other nine models on the GC10-DET verification data set. From the table, it can be seen that, corresponding to mAP50, the improved network model has the highest score of 95.3%, which is 10.5% higher than the second highest score; for mAP50:95, the improved network model is the highest at 72.8%, and YOLOv10m is the second highest at 54.0%, which is 18.8% higher than the improved network model. Although not the first in FPS compared with other excellent algorithms in the table, the improved network model still has the highest detection accuracy while maintaining high real-time performance. At the same time, the above experimental results obtained on the GC10-DET data set further show that the improved model has good generalization performance and effectiveness.

[0121] Table 3 shows the comparison of AP50 experimental results of each defect category of the model with other models on the GC10-DET verification data set. There are 11 rows of data in the table, and the first row represents the average AP50 value of all categories, and the remaining 10 rows represent the AP50 value of each category. According to the data in Table 3, the following Figure 11 is created to visualize the differences between models. According to Figure 11 and each row of data in Table 3, it can be seen that the improved model has the best detection effect in all defect categories, and the average detection accuracy of the improved model is much higher than that of other mainstream models in the table, which further verifies the superiority of the improved model in defect detection performance.

[0122] In one embodiment, based on the pre-constructed improved model, the surface defects of the metal to be detected are detected in real time, including:

[0123] Step S11, acquiring a surface image of the metal to be detected;

[0124] Step S12, based on the improved model, detecting the surface defects of the metal to be detected in real time according to the surface image.

[0125] The working principle and beneficial effects of the above technical solutions are:

[0126] Based on the improved model, when performing real-time detection of surface defects of the to-be-detected metal, a surface image of the to-be-detected metal is acquired, and the surface image is input into the improved model, and the improved model automatically performs surface defect analysis according to the surface image.

[0127] In one embodiment, the surface image of the to-be-detected metal is acquired by:

[0128] Step S111, when the user enters the metal stacking site, control the unmanned aerial vehicle to acquire the metal use intention prediction basis of the user; wherein the metal use intention prediction basis at least includes: the image of the metal selected by the user from the metal stacking site, the work image of the user;

[0129] In step S111, a large number of metals with surface defects are stacked in the metal stacking site; when the user enters the metal stacking site, it is hoped to find metals with surface defects within the user's acceptance range for use; the unmanned aerial vehicle is always resident in the metal stacking site. Therefore, when the user enters the metal stacking site, the unmanned aerial vehicle is controlled to acquire the metal use intention prediction basis of the user; among them, the metal selected by the user from the metal stacking site refers to the metal that the user has selected and wants to use, and the image acquisition can be controlled by the unmanned aerial vehicle to take pictures; the work image of the user can be a metal recycling task, a preferred metal type, etc.;

[0130] Step S112, when predicting the first metal use intention and the second metal use intention of the user based on the metal use intention prediction basis; wherein the first metal use intention at least includes: metal size, metal type, metal shape; the second metal use intention at least includes: acceptable metal surface defects;

[0131] In step S112, the metal use intention prediction basis will continuously reflect which metal with surface defects the user wants, so the first metal use intention and the second metal use intention of the user can be predicted based on the metal use intention prediction basis;

[0132] Step S113, control the unmanned aerial vehicle to search for a plurality of target metals in the metal stacking site that match the first metal use intention, and take the target metals as to-be-detected metals;

[0133] In step S113, when controlling the unmanned aerial vehicle to search, the unmanned aerial vehicle can be controlled to fly in the metal stacking site to search;

[0134] Step S114, control the unmanned aerial vehicle to perform surface shooting on the to-be-detected metal to obtain a surface image.

[0135] In step S114, after searching for the target metal and as the metal to be detected, the unmanned aerial vehicle is controlled to take a surface shot of the metal to be detected.

[0136] The embodiment of the present application is the acquisition of the surface image in the special case that the user enters the metal stacking site to find the metal with surface defects within his own acceptance range for use.

[0137] In one embodiment, after the unmanned aerial vehicle is controlled to take a surface shot of the metal to be detected and the surface image is obtained, the method further comprises:

[0138] In step S115, when the detection result of the real-time detection of the surface defects of the metal to be detected based on the improved model according to the surface image is consistent with the second metal use intention, the unmanned aerial vehicle is controlled to prompt the user to use the corresponding metal to be detected.

[0139] In step S115, after the surface image is obtained, the improved model analyzes the surface image to determine the surface defect formation detection result; when the detection result is consistent with the second metal use intention, it means that the corresponding metal to be detected will be used by the user, and the unmanned aerial vehicle is controlled to prompt the user to use the corresponding metal to be detected. When prompting, a light ring can be projected onto the corresponding metal to be detected by the light ring projection device on the unmanned aerial vehicle, so that the user can quickly find it.

[0140] The embodiment of the present application is aimed at the special case that the user enters the metal stacking site to find the metal with surface defects within his own acceptance range for use. By controlling the unmanned aerial vehicle, the user can quickly find the metal within his own acceptance range, greatly improving the user's work efficiency.

[0141] In one embodiment, the control of the unmanned aerial vehicle to take a surface shot of the metal to be detected comprises:

[0142] In step S1141, the distribution of the metal to be detected is determined based on the stacking position of the metal to be detected in the metal stacking site.

[0143] In step S1141, after the unmanned aerial vehicle searches for the metal to be detected, it also acquires the stacking position of the metal to be detected in the metal stacking site, and the stacking positions of the plurality of metals to be detected form the distribution of the metal to be detected.

[0144] In step S1142, the metal cluster region is determined in the distribution of the metal to be detected based on the metal cluster region condition. The metal cluster region condition at least includes that the straight line distance between the stacking positions of any two metals to be detected in the same metal cluster region is less than a first distance threshold.

[0145] In step S1142, the metal cluster region refers to a region containing a plurality of stacking positions in the metal distribution to be detected; the first distance threshold can be 3 meters; and the straight-line distance between the stacking positions of the metals to be detected in the same metal cluster region is less than the first distance threshold, which can ensure that the UAV can continuously take surface photos of all the metals to be detected in the same metal cluster region one by one in proximity.

[0146] In step S1143, the UAV is controlled to obtain a historical movement trajectory and a historical clear visual range of the user in a recent preset time period.

[0147] In step S1143, the recent preset time period refers to a time period in the last 5 minutes; the historical movement trajectory is the movement trajectory of the user in the metal stacking site in the recent preset time period, which can be obtained based on the UAV; and the historical clear visual range is the clear visual range of the user in the metal stacking site in the recent preset time period, which can be determined by the UAV according to the eye orientation of the user, the normal naked-eye clear visual distance, and the like.

[0148] In step S1144, a target metal cluster region that meets a standard relative relationship with the historical movement trajectory and the historical clear visual range is determined from the metal cluster regions.

[0149] In step S1144, when the target metal cluster region meets the standard relative relationship with the historical movement trajectory and the historical clear visual range, it indicates that the surface photographing of the target metal cluster region by the UAV will not affect the current searching work of the user, and if the target metal is searched in the target metal cluster region, it is convenient for the user to quickly pick and sort the target metal, and the like.

[0150] In step S1145, the UAV is controlled to go to the target metal cluster region to take surface photos of the corresponding metals to be detected.

[0151] The standard relative relationship at least includes:

[0152] If the proportion difference between the first local trajectory proportion of the historical movement trajectory falling into the first semicircular region and the second local trajectory proportion of the historical movement trajectory falling into the second semicircular region exceeds a proportion threshold, and the overlapping area of the metal cluster region and the second semicircular region exceeds an area threshold.

[0153] If the difference in the proportion is not more than the proportion threshold value, the metal cluster region does not overlap with the first semicircular region and the second semicircular region, respectively, and the closest distance between the metal cluster region and the first semicircular region or the second semicircular region is not more than the second distance threshold value; wherein the target line segment with the history moving track connected at the head and the tail is taken as the diameter to make a circular region, the target line segment divides the circular region into the first semicircular region and the second semicircular region; the closest distance is the smaller one of the distance between the metal cluster region and the first semicircular region and the distance between the metal cluster region and the second semicircular region;

[0154] The metal cluster region has not historically entered the historical clear visible range completely;

[0155] The overlapping degree of the region in which the metal cluster region has historically entered the historical clear visible range partially is not more than the overlapping degree threshold value.

[0156] In step S1145, after the target metal cluster region is determined, the unmanned aerial vehicle is controlled to go to the target metal cluster region to take a surface shot of the corresponding metal to be detected, so as to ensure the suitability of the unmanned aerial vehicle in shooting the metal to be detected to obtain a surface image.

[0157] The historical moving track is connected at both ends to form a target line segment. When a circular region is made with the target line segment as a diameter, the midpoint of the target line segment is the center of the circle, and a circular region is made with half the length of the target line segment as a radius. The first local track proportion is the ratio of the length of the local track in which the historical moving track falls within the first semicircular region to the total length of the historical moving track. The second local track proportion is the ratio of the length of the local track in which the historical moving track falls within the second semicircular region to the total length of the historical moving track. The proportion difference refers to the difference between the first local track proportion and the second local track proportion. The party with the larger proportion value is taken as the first local track proportion, and the party with the smaller proportion value is taken as the second local track proportion. The proportion threshold value can be 3 / 10. The area threshold value can be 1.5 square meters. The second distance threshold value can be 2 meters. If the proportion difference between the first local track proportion in which the historical moving track falls within the first semicircular region and the second local track proportion in which the historical moving track falls within the second semicircular region exceeds the proportion threshold value, it indicates that the user's moving range in the recent preset time period involves both the first semicircular range and the second semicircular range, but involves the first semicircular range relatively more. At this time, the overlapping area of the metal cluster region and the second semicircular region is ensured to exceed the area threshold value. If the unmanned aerial vehicle enters the metal cluster region, it will not affect the user's current search for work. If the target metal is found in the target metal cluster region, it is also convenient for the user to quickly pick and sort the target metal, etc. If the proportion difference does not exceed the proportion threshold value, it indicates that the user's moving range in the recent preset time period involves both the first semicircular range and the second semicircular range, and the involvement degree is similar. At this time, the metal cluster region is ensured not to overlap with the first semicircular region and the second semicircular region, respectively, and the closest distance between the metal cluster region and the first semicircular region or the second semicircular region does not exceed the second distance threshold value. If the unmanned aerial vehicle enters the metal cluster region, it will not affect the user's current search for work. If the target metal is found in the target metal cluster region, it is also convenient for the user to quickly pick and sort the target metal, etc. The overlap degree refers to the overlap degree between the regions in which the metal cluster region has locally entered the historical clear visibility range in the past. The overlap degree threshold value can be 20%. The metal cluster region is ensured not to have completely entered the historical clear visibility range in the past, and the overlap degree of the region in which the metal cluster region has locally entered the historical clear visibility range does not exceed the overlap degree threshold value. The metal cluster region can be constrained to be a region that the user has not completely and clearly seen but may want to search. If the unmanned aerial vehicle enters the metal cluster region, it will not affect the user's current search for work. If the target metal is found in the target metal cluster region, it is also convenient for the user to quickly pick and sort the target metal, etc.

[0158] When the unmanned aerial vehicle takes surface pictures of the metal to be detected, it may affect the current work of the user, and if the target metal is found in the target metal cluster area, the user cannot quickly pick the target metal. To solve these problems, the embodiment of the present application introduces a standard relative relationship to determine the target metal cluster area that meets the standard relative relationship between the historical moving track and the historical clear visual range from the metal cluster area, controls the unmanned aerial vehicle to take surface pictures of the corresponding metal to be detected in the target metal cluster area, and if the unmanned aerial vehicle enters the metal cluster area, it will not affect the current work of the user, and if the target metal is found in the target metal cluster area, it is convenient for the user to quickly pick the target metal, greatly improving the rationality, applicability and accuracy of the system in obtaining surface images.

[0159] The embodiment of the present application provides a metal surface defect real-time detection system based on an improved model, which comprises:

[0160] A detection module is configured to perform real-time detection of surface defects of the metal to be detected based on the pre-constructed improved model.

[0161] The pre-construction step of the improved model comprises:

[0162] First model improvement: replace the C2f module of the preset part in the YOLOv8m model with the C2f_ASCF module and the C2f_E-ASCF module;

[0163] Second model improvement: add the FFDPN module to the Neck part in the YOLOv8m model;

[0164] Third model improvement: replace the detection head of the YOLOv8m model with the LE-SCDH detection head;

[0165] The YOLOv8m model after the first model improvement, the second model improvement and the third model improvement is trained using a data set labeled with metal surface defects to obtain the improved model.

[0166] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include these modifications and variations.

Claims

1. A real-time detection method for metal surface defects based on an improved model, characterized in that, include: When a user enters the metal storage area, the drone is controlled to obtain the user's metal usage intention prediction basis; the metal usage intention prediction basis includes at least: images of the metal picked by the user from the metal storage area and the user's work profile. Based on the prediction criteria for metal usage intention, the user's first metal usage intention and second metal usage intention are predicted; wherein, the first metal usage intention includes at least: metal size, metal type, and metal shape; the second metal usage intention includes at least: acceptable metal surface defects; The drone is controlled to search for multiple target metals that match the intended use of the first metal within the metal storage area, and these are designated as the metals to be detected. The distribution of the metal to be tested is determined based on its location within the metal storage area. Based on the metal cluster region condition, a metal cluster region is determined in the distribution of metals to be detected; wherein, the metal cluster region condition includes at least the following: the straight-line distance between the stacking positions of two metals to be detected in the same metal cluster region is less than a first distance threshold. Control the drone to obtain the user's historical movement trajectory and historical clear visible range within the most recent preset time period; Identify target metal cluster regions within the metal cluster regions that conform to a standard relative relationship with historical movement trajectories and historical clearly visible ranges. Control the drone to travel to the target metal cluster area and take surface photos of the corresponding metal to be detected; The standard relative relationships include at least the following: If the proportion of the first partial trajectory falling into the first semicircular area of ​​the historical movement trajectory exceeds the proportion of the second partial trajectory falling into the second semicircular area, and the difference in proportion exceeds the proportion threshold, the area of ​​the overlapping area between the metal cluster area and the second semicircular area exceeds the area threshold. If the proportion difference does not exceed the proportion threshold, the metal cluster region does not overlap with either the first semicircular region or the second semicircular region, and the closest distance between the metal cluster region and either the first or second semicircular region does not exceed the second distance threshold. Specifically, a circular region is drawn with the target line segment connecting the beginning and end of the historical movement trajectory as the diameter, and the target line segment divides the circular region into the first semicircular region and the second semicircular region. The closest distance is the smaller of the distance between the metal cluster region and the first semicircular region and the distance between the metal cluster region and the second semicircular region. The metal cluster area has never been fully included in the historically clear visible range. The overlap of areas in the metal cluster region that have historically entered the clearly visible range does not exceed the overlap threshold. Based on the improved model, surface defects of the metal to be inspected are detected in real time according to the surface image. The pre-construction steps for improving the model include: The first model improvement is to replace all the C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules; A second model improvement was made: the FFDPN module was added to the Neck part of the YOLOv8m model; The third model improvement involves replacing the detector head of the YOLOv8m model with the LE-SCDH detector head. Using a dataset labeled with metal surface defects, the YOLOv8m model, after undergoing the first, second, and third model improvements, was trained to obtain an improved model.

2. The real-time detection method for metal surface defects based on the improved model as described in claim 1, characterized in that, The C2f_ASCF module is the C2f module after all Bottleneck modules have been replaced with the Bottleneck_ASCF module; Bottleneck_ASCF includes the ASCF module; the ASCF module divides the input feature map into four groups, two of which are convolutionally operated on using different convolution kernels, while the other two remain unchanged. After the convolution operation is completed, the four groups of feature maps are concatenated by the Concat module, and then further fused by 1x1 convolution to achieve efficient integration of channel features, and finally generate an output feature map with rich contextual information.

3. The real-time detection method for metal surface defects based on the improved model as described in claim 1, characterized in that, The C2f_E-ASCF module is the C2f module after all the Bottleneck modules have been replaced with the Bottleneck_E-ASCF module. The E-ASCF module divides the input feature map into four groups and applies 1x1, 3x3, 5x5 and 7x7 convolution operations to each group respectively. The four groups of feature maps are merged, and then the channel features of each group are fused by 1x1 convolution to generate the final output feature map.

4. The real-time detection method for metal surface defects based on the improved model as described in claim 1, characterized in that, The FFDPN module consists of a FeatureFocus module and a feature diffusion mechanism. The FeatureFocus module receives inputs at three different scales and adopts an Inception-style design, using parallel deep convolutions to accurately select and aggregate rich information from multiple scales. The FeatureFocus module aligns high-dimensional and low-dimensional features with the features of the current layer through convolution and segmentation operations. Then, these processed feature maps are concatenated to form a comprehensive feature map that integrates multi-level information. The comprehensive feature map is input into a set of parallel depthwise separable convolutions for contextual feature extraction. Next, the fused multi-scale features are further processed through 1×1 convolutions to finally output the feature map.

5. The real-time detection method for metal surface defects based on the improved model as described in claim 1, characterized in that, The LE-SCDH detection head uses a shared convolution strategy to allow the output of the same convolutional layer to be used for multiple prediction tasks simultaneously. The LE-SCDH detection head uses 1×1 convolution and group normalization to reduce the dimensionality of the feature map, which not only reduces the computational load and the risk of overfitting, but also ensures that the most critical feature information is preserved. The LE-SCDH detector uses a first 3×3 shared convolutional layer to capture spatial information and a second 3×3 shared convolutional layer to focus on semantic information. The scale layer in the LE-SCDH detector head addresses inconsistencies in multi-scale target detection by applying a scaling factor to each element within a channel.

6. The real-time detection method for metal surface defects based on the improved model as described in claim 1, characterized in that, After controlling the drone to take surface photos of the metal to be inspected and obtaining surface images, the process also includes: When the detection result of real-time surface defect detection of the metal to be detected based on the improved model and the surface image matches the intended use of the second metal, the control drone prompts the user to use the corresponding metal to be detected.

7. A real-time detection system for metal surface defects based on an improved model, characterized in that, include: The detection module is used for: When a user enters the metal storage area, the drone is controlled to obtain the user's metal usage intention prediction basis; the metal usage intention prediction basis includes at least: images of the metal picked by the user from the metal storage area and the user's work profile. Based on the prediction criteria for metal usage intention, the user's first metal usage intention and second metal usage intention are predicted; wherein, the first metal usage intention includes at least: metal size, metal type, and metal shape; the second metal usage intention includes at least: acceptable metal surface defects; The drone is controlled to search for multiple target metals that match the intended use of the first metal within the metal storage area, and these are designated as the metals to be detected. The distribution of the metal to be tested is determined based on its location within the metal storage area. Based on the metal cluster region condition, a metal cluster region is determined in the distribution of metals to be detected; wherein, the metal cluster region condition includes at least the following: the straight-line distance between the stacking positions of two metals to be detected in the same metal cluster region is less than a first distance threshold. Control the drone to obtain the user's historical movement trajectory and historical clear visible range within the most recent preset time period; Identify target metal cluster regions within the metal cluster regions that conform to a standard relative relationship with historical movement trajectories and historical clearly visible ranges. Control the drone to travel to the target metal cluster area and take surface photos of the corresponding metal to be detected; The standard relative relationships include at least the following: If the proportion of the first partial trajectory falling into the first semicircular area of ​​the historical movement trajectory exceeds the proportion of the second partial trajectory falling into the second semicircular area, and the difference in proportion exceeds the proportion threshold, the area of ​​the overlapping area between the metal cluster area and the second semicircular area exceeds the area threshold. If the proportion difference does not exceed the proportion threshold, the metal cluster region does not overlap with either the first semicircular region or the second semicircular region, and the closest distance between the metal cluster region and either the first or second semicircular region does not exceed the second distance threshold. Specifically, a circular region is drawn with the target line segment connecting the beginning and end of the historical movement trajectory as the diameter, and the target line segment divides the circular region into the first semicircular region and the second semicircular region. The closest distance is the smaller of the distance between the metal cluster region and the first semicircular region and the distance between the metal cluster region and the second semicircular region. The metal cluster area has never been fully included in the historically clear visible range. The overlap of areas in the metal cluster region that have historically entered the clearly visible range does not exceed the overlap threshold. Based on the improved model, surface defects of the metal to be inspected are detected in real time according to the surface image. The pre-construction steps for improving the model include: The first model improvement is to replace all the C2f modules in the preset part of the YOLOv8m model with C2f_ASCF modules and C2f_E-ASCF modules; A second model improvement was made: the FFDPN module was added to the Neck part of the YOLOv8m model; The third model improvement involves replacing the detector head of the YOLOv8m model with the LE-SCDH detector head. Using a dataset labeled with metal surface defects, the YOLOv8m model, after undergoing the first, second, and third model improvements, was trained to obtain an improved model.

Images