A small target detection method for surface defects based on multi-scale feature interaction

Abstract

The present invention discloses a method for detecting small targets with surface defects based on multi-scale feature interaction. The method comprises: inputting a surface image of a product to be detected, which is collected in real time, into a detection model; after the detection model processes the surface image, it outputs an image with the possible defect classification category, bounding box coordinates and confidence level of the defect target; wherein the detection model is a model based on the YOLOv11 network, comprising a backbone network, a neck network and a detection head connected in series; a multi-scale feature extraction SMK module and an attention mechanism SELA‑S module are introduced into the backbone network; and two feature fusion Fafusion modules are introduced into the neck network. The present invention can effectively extract and fuse multi-scale target features under complex backgrounds, and has good processing performance for distinguishing defects from complex backgrounds in images and detecting multi-scale targets. Moreover, it can focus on defect areas more accurately in surface defect detection tasks.

Description

Technical Field

[0001] The present invention relates to the field of defect detection technology, and in particular to a method for detecting small targets of surface defects based on multi-scale feature interaction. Background Art

[0002] Traditional methods for steel defect detection primarily involve manual visual inspection and contact testing techniques. However, these traditional methods have numerous limitations. Manual inspection is not only inefficient but also subject to human error, making it prone to misjudgments and missed detections. While contact testing can reveal some hidden defects, it is complex, requires long inspection cycles, and often requires pre-treatment of the steel surface, potentially damaging the steel. Furthermore, these traditional methods are labor-intensive and cannot withstand the long, intensive workloads required. Furthermore, they cannot meet the high precision and efficiency requirements of large-scale production processes, such as assembly lines.

[0003] Compared with traditional methods, machine learning methods have higher efficiency and stability in steel defect detection, can automatically analyze large-scale data and reduce human errors. However, this method still has some limitations. First, traditional machine learning methods usually rely on manually designed features, but when faced with defects with irregular shapes and blurred boundaries, it is difficult to effectively extract key features, resulting in missed detections or false detections. Secondly, machine learning methods usually have poor environmental adaptability and are difficult to adapt to the environment of steel manufacturing plants. Furthermore, some machine learning models consume large amounts of computing resources and have limited real-time performance, making it difficult to meet the needs of practical applications.

[0004] In recent years, deep learning, particularly convolutional neural networks (CNNs), has demonstrated outstanding performance in steel surface defect detection. Traditional detection methods, including manual inspection and machine learning, cannot meet the high precision and efficiency requirements. The introduction of deep learning technology has greatly promoted the development of steel surface defect detection. In particular, the YOLOv11 algorithm, the benchmark model selected in this article, has become a popular choice in the industry due to its efficient processing speed and high detection accuracy. However, YOLOv11 can still miss or misidentify some complex or subtle defects in steel surface defect detection applications. Summary of the Invention

[0005] In order to solve the technical problem of easy missed detection or false detection in the current surface defect detection for complex or subtle defects, the present invention provides a small target detection method for steel surface defects based on multi-scale feature interaction, which can effectively improve the detection accuracy of steel surface defects in the steel production process.

[0006] In order to achieve the above technical objectives, the technical solution of the present invention is:

[0007] A method for detecting small targets of surface defects based on multi-scale feature interaction includes the following steps:

[0008] Step 1: Input the surface image of the product to be inspected, which is collected in real time, into the inspection model;

[0009] Step 2: After the detection model processes the surface image, it outputs an image with the possible defect classification category, bounding box coordinates, and confidence level of the defect target;

[0010] The detection model is based on the YOLOv11 network as the basic structure, including the backbone network, neck network and detection head connected in series.

[0011] The backbone network introduces a multi-scale feature extraction SMK module and an attention mechanism SELA-S module; the neck network introduces two feature fusion Fafusion modules;

[0012] The SMK module is set between the third and fourth C3K2 modules of the backbone network, and includes a residual structure path, a local information extraction path, and a spatial information extraction path for processing the feature map input to the SMK module respectively. The processing results of the three paths are spliced ​​as output;

[0013] The SELA-S module is set between the original SPPF module and the C2PSA module of the backbone network, and includes a spatial attention mechanism pathway and a channel attention mechanism pathway for processing the feature maps input to the SELA-S module respectively. The processing results of the two pathways are multiplied by the input feature maps respectively, and the multiplication results are added to the input feature maps as output;

[0014] The two Fafusion modules are connected in series, wherein the first Fafusion module is arranged after the first upsampling module of the neck network, and the output is processed by the first C3K2 module and the second upsampling module of the neck network as the input of the second Fafusion module; both Fafusion modules include four paths for processing feature maps, and the processing results of the four paths are spliced ​​and the channels are adjusted as output.

[0015] Furthermore, in step 1, the detection model is trained by the following steps:

[0016] Step 101: Obtain surface images of products with surface defects, annotate them and use them as a dataset, then perform data augmentation on the images in the dataset to expand the dataset.

[0017] In step 102, the expanded data set is divided into a training set, a validation set, and a test set according to a preset ratio. The training set is then input into the detection model for iterative training. During the iterative training process, the validation set is periodically input into the detection model to adjust the model training strategy according to the verification effect. Finally, after the iterative training is completed, the training effect is verified and the performance of the detection model is evaluated by inputting the test set.

[0018] Furthermore, in step 101, the data enhancement is to perform processing on the image including flipping, rotation, cropping and brightness adjustment, so as to generate a processed new image based on the original image to expand the data set.

[0019] Furthermore, in step 102, during the iterative training process, multi-scale features are extracted through the backbone network, and then the neck network realizes cross-layer feature interaction and fusion, and finally the detection head outputs the classification category, bounding box coordinates and target existence confidence as the result; the result is then compared with the label value, and back propagation is performed to update the parameters of the detection model; at the same time, the verification set is regularly input into the detection model for verification, so as to further adjust the parameters of the detection model according to the verification results; finally, after verifying the training effect through the test set and evaluating the performance of the detection model, the parameters of the detection model obtained by training are saved.

[0020] Furthermore, the backbone network of the detection model includes a twelve-layer module structure connected in series, wherein the first layer is the first Conv module, the second layer is the second Conv module, the third layer is the first C3K2 module, the fourth layer is the third Conv module, the fifth layer is the second C3K2 module, the sixth layer is the fourth Conv module, the seventh layer is the third C3K2 module, the eighth layer is the SMK module, the ninth layer is the fourth C3K2 module, the tenth layer is the SPPF module, the eleventh layer is the SELA-S attention mechanism module, and the twelfth layer is the C2PSA module.

[0021] Furthermore, the SMK module includes:

[0022] The residual structure path includes a 1×1 Conv module;

[0023] The local information extraction pathway includes a Conv module and a channel attention mechanism module connected in series;

[0024] The spatial information extraction pathway includes two sub-pathways of different scales. One of them includes a 1×1 Conv module, a 3×3 Conv module, a dilated convolution module with a dilation of 2 in the x-direction, and a dilated convolution module with a dilation of 2 in the y-direction, which are connected in series. The other sub-pathway includes a 1×1 Conv module, a 5×5 Conv module, a dilated convolution module with a dilation of 3 in the x-direction, and a dilated convolution module with a dilation of 3 in the y-direction, which are connected in series.

[0025] Concat module for concatenating the residual structure pathway, local information extraction pathway, and spatial information extraction pathway.

[0026] Furthermore, the SELA-S attention mechanism module includes:

[0027] The spatial attention mechanism pathway includes two sets of spatial feature weight extraction modules connected in series. Each set of modules includes a connected average pooling layer, a feature extraction layer, group normalization, and a sigmoid function. The average pooling layer of one set of modules is a vertical average pooling layer, and the average pooling layer of the other set is a horizontal average pooling layer. The output of the first set of modules is multiplied by the feature map of the input SELA-S attention mechanism module and then input into the second set of modules to obtain spatial feature weights in two different directions.

[0028] The channel attention mechanism pathway includes an adaptive pooling layer and a convolutional layer connected in series, and finally passes through a sigmoid function to obtain the channel feature weights;

[0029] The processing results of the two pathways, including the spatial feature weights in the vertical and horizontal directions, and the channel feature weights obtained by the channel attention mechanism pathway, are then multiplied with the original feature map respectively. Finally, each result is added to the original feature map as the residual branch as the output.

[0030] Furthermore, the neck network of the detection model includes a twelve-layer module structure connected in series, wherein the first layer is a first upsampling module, the second layer is a first Fafusion module, the third layer is a first C3K2 module, the fourth layer is a second upsampling module, the fifth layer is a second Fafusion module, the sixth layer is a second C3K2 module, the seventh layer is a first Conv module, the eighth layer is a first Concat module, the ninth layer is a third C3K2 module, the tenth layer is a second Conv module, the eleventh layer is a second Concat module, and the twelfth layer is a fourth C3K2 module;

[0031] The input end of the first Fafusion module is also connected to the second C3K2 module, the third C3K2 module and the fourth C3K2 module of the backbone network respectively; the input end of the second Fafusion module is also connected to the first C3K2 module, the second C3K2 module and the third C3K2 module of the backbone network respectively; the output end of the second C3K2 module is also connected to the detection head; the input end of the first Concat module is also connected to the first C3K2 module of the neck network; the output end of the third C3K2 module is also connected to the detection head; the input end of the second Concat module is also connected to the C2PSA module of the backbone network.

[0032] Furthermore, the Fafusion module in the neck network has four pathways including: a large-size feature map downsampling pathway, including DepthwiseConv and PointwiseConv connected in series; a medium-size feature map channel number adjustment pathway, including a Conv module for adjusting the number of feature map channels; two small-size feature map pathways, each including an Unsample module. Finally, the output images of the four pathways are concat- ored and then passed through the Conv adjustment pathway to obtain the output feature map.

[0033] The technical effects of the present invention are:

[0034] Based on the existing YOLOv11 network, the present invention introduces the SMK module to replace the Conv module in the original network model, which can enable the model to effectively extract and fuse multi-scale target features in complex backgrounds. At the same time, the parameter calculation amount is lower than that of the Conv module, and it has better processing performance for distinguishing defects from complex backgrounds in steel images and multi-scale target detection.

[0035] At the same time, the present invention also introduces the SELA-S attention mechanism, which enables the model to focus on the defect area more accurately in the steel surface defect detection task, while suppressing irrelevant background information, improving feature expression capabilities, and thus improving detection accuracy and generalization capabilities.

[0036] The present invention also enhances the network's information interaction capability at different scale feature levels through a multi-scale feature fusion pyramid network and a feature fusion module Fafusion, ensuring global information sharing between different scale features of the network. The model thus formed has a better detection effect.

[0037] When training the model, this paper uses the Focaler-PIoUv2 bounding box loss function, enabling the model to focus on different regression samples for different detection tasks, while ensuring more accurate and stable bounding box regression. Furthermore, when obtaining the training dataset, data augmentation is used to preprocess the dataset, effectively expanding the morphological diversity of the defect samples. This data augmentation mechanism mitigates the risk of overfitting under limited sample conditions, thereby improving the model's generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the small target detection model for steel surface defects based on the multi-scale feature interaction of the technical solution of the present invention.

[0039] Figure 2 This is a schematic diagram of the SMK module in the technical solution of the present invention.

[0040] Figure 3 This is a schematic diagram of the Fafusion module in the technical solution of the present invention.

[0041] Figure 4 Schematic diagram of the SELA-S module in the technical solution of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific implementation plans.

[0043] Example 1:

[0044] like Figure 1 As shown, the present embodiment provides a detection method for a small target detection model of steel surface defects based on multi-scale feature interaction, including:

[0045] The surface image of the product to be inspected, acquired in real time, is input into the inspection model.

[0046] The detection model is based on the YOLOv11 network as the basic structure, including the backbone network, the neck network and the detection head connected in series. This embodiment introduces the multi-scale feature extraction module SMK module and the SELA-S attention mechanism module into the YOLOv11 network, and then introduces the multi-scale feature fusion pyramid network and the feature fusion module Fafusion, thereby obtaining the multi-scale feature interaction steel surface defect small target detection model used in this embodiment. The model structure is as follows: Figure 2 shown.

[0047] The backbone network in this embodiment introduces a multi-scale feature extraction SMK module and an attention mechanism SELA-S module. The backbone network includes a twelve-layer module structure connected in series, wherein the first layer is the first Conv module, the second layer is the second Conv module, the third layer is the first C3K2 module, the fourth layer is the third Conv module, the fifth layer is the second C3K2 module, the sixth layer is the fourth Conv module, the seventh layer is the third C3K2 module, the eighth layer is the SMK module, the ninth layer is the fourth C3K2 module, the tenth layer is the SPPF module, the eleventh layer is the SELA-S attention mechanism module, and the twelfth layer is the C2PSA module.

[0048] In this embodiment, the Conv module includes a standard two-dimensional convolution layer, a BatchNormalization layer, and an activation function SILU layer connected in series.

[0049] The C3K2 module of this embodiment includes a Conv module and a C3K module, and the C2PSA module includes a Conv module and a PSA module connected in series.

[0050] Traditional ordinary Conv only uses a fixed convolution kernel size (for example, 3×3) and can only perceive features of fixed size, and can only learn patterns under single-scale features. It may ignore some key defect information, making it difficult for the model to capture small scratches and large-area defects at the same time. Therefore, the SMK module is introduced in this embodiment. The SMK module of this embodiment refers to the Scale Multi-path Kernel fusion module. The SMK module is arranged between the original third and fourth C3K2 modules of the backbone network, and includes a residual structure path, a local information extraction path, and a spatial information extraction path for processing the feature maps input into the SMK module respectively. The processing results of the three paths are spliced ​​as output.

[0051] See also Figure 3 The SMK module extracts multi-scale features from images, processes each scale feature differently, and finally fuses the obtained multi-scale features, thereby enhancing the model's multi-scale feature extraction and fusion capabilities, effectively enhancing the perception of multi-scale defect targets in complex backgrounds. The SMK module improves the detection accuracy of steel surface defects through multi-scale convolution combination, while reducing the computational complexity of the model, enabling it to have higher computational efficiency while ensuring detection performance. In addition, the SMK module can fully distinguish defect areas from complex background information, improve the model's robustness to defect targets of different scales, and thus demonstrate better adaptability in steel defect detection tasks.

[0052] Among the three main pathways of the SMK module, the residual structure pathway includes a 1×1 Conv module. The local information extraction pathway consists of a series of Conv modules and a channel attention mechanism module. The spatial information extraction pathway includes two sub-pathways of different scales. One consists of a series of 1×1 Conv modules, 3×3 Conv modules, a dilated convolution module with a dilation of 2 in the x-direction, and a dilated convolution module with a dilation of 2 in the y-direction; the other consists of a series of 1×1 Conv modules, 5×5 Conv modules, a dilated convolution module with a dilation of 3 in the x-direction, and a dilated convolution module with a dilation of 3 in the y-direction. The SMK module also includes a Concat module at the end for splicing the residual structure pathway, the local information extraction pathway, and the spatial information extraction pathway. The final output feature map is obtained by concatenating the feature map processed by the residual structure pathway with the feature maps processed by the local information extraction pathway and the spatial information extraction pathway.

[0053] For ordinary convolution, if the input feature map size is H×W×C1, that is, the height is H, the width is W, and the number of channels is C1. The convolution kernel size of ordinary convolution is 3×3. If the output feature map size is H out ×W out ×C2, the calculation formulas for the parameter amount Conv Params and the computation amount Conv FLOPs are as follows:

[0054] Conv Params = 9 × C1 × C2;

[0055] Conv FLOPs = 9 × C1 × C2 × H out ×W out ;

[0056] For the SMK module, the size of the input feature is also H×W×C, and the parameter calculation formula is as follows:

[0057]

[0058] The calculation formula for the amount of calculation is as follows:

[0059]

[0060] If C1=C2=C, then the number of parameters is Conv Params=9C 2 ,SMK Params≈4.63C 2 +1.25C, the number of parameters is reduced by about 48.5%, and the computational cost is Conv FLOPs = 9C 2 ×H out ×W out, SMK FLOPs≈4.56H out E out C 2 +0.06C 2 , reducing computational complexity by approximately 49%. In the steel surface defect detection scenario where features have varying sizes, shapes, and textures, this embodiment introduces the SMK module, which not only enhances model detection efficiency and reduces model complexity, but also improves network generalization and detection accuracy.

[0061] The SELA-S module of this embodiment is arranged between the original SPPF module and the C2PSA module of the backbone network, and includes a spatial attention mechanism pathway and a channel attention mechanism pathway for processing the feature maps input into the SELA-S module respectively. The processing results of the two pathways are multiplied by the input feature maps respectively, and the multiplication results are added to the input feature maps as output.

[0062] Specifically, the spatial attention mechanism pathway in the SELA-S attention mechanism module of this embodiment includes two sets of spatial feature weight extraction modules connected in series. Each set of modules includes a connected average pooling layer, a feature extraction layer, group normalization, and a Sigmoid function. The average pooling layer of one set of modules is a vertical average pooling layer, and the other set is a horizontal average pooling layer. The output of the first set of modules is multiplied by the feature map input to the SELA-S attention mechanism module and then input to the second set of modules. That is, the spatial attention mechanism includes two serial average pooling layers and feature extraction layers divided into two different directions, vertical and horizontal, to obtain spatial feature weights in two different directions.

[0063] The channel attention mechanism pathway includes adaptive pooling layers and convolution layers connected in series, and finally passes through the sigmoid function to obtain the channel feature weights.

[0064] Finally, the processing results of the two pathways, including the spatial feature weights in the vertical and horizontal directions, and the channel feature weights obtained by the channel attention mechanism pathway, are multiplied with the original feature map respectively, and each result is added to the original feature map as the residual branch as the output.

[0065] The specific implementation of the channel attention mechanism is as follows: the input feature map X with size H×W×C c Perform global average pooling to generate a 1×1×C feature vector Z C , to obtain a channel-level statistic with a global receptive field of channel C:

[0066]

[0067] Where P represents the global average pooling operation, i represents the row index (from 1 to H), and j represents the row index (from 1 to W).

[0068] Then learn the dependencies between channels through a single layer of convolution:

[0069] s=σ(F c ·Z c ));

[0070] Among them, s is the channel weight, F c Represents the 1x1 convolution kernel parameter, and σ represents the nonlinear activation function Sigmoid.

[0071] The other spatial attention path first average pools each channel c column by column in the width W direction to generate horizontal features The formula is as follows:

[0072]

[0073] Among them, W represents the width of the feature map, H represents the height of the feature map, h represents the height position index of the feature map, and the value range is 0≤h≤H, i represents the width position index of the feature map, and the value range is 0≤i≤W, x c (h,i) represents the eigenvalue of channel c at position (h,i), that is, the value of the feature map at height h, width i, and channel c.

[0074] Then, a set of depth-wise separable convolutions including depth-wise convolution DWConv and point-wise convolution PWConv are used to further extract features, and then group normalization Gn is used to enhance position information. Finally, the horizontal spatial attention weight y is obtained through the Sigmoid activation function σ. h The formula is as follows:

[0075]

[0076] Then combine the obtained weight with the original feature map X c Combined to obtain feature map F2

[0077] F2=X c ×y h ;

[0078] Then, based on the feature map F2, average pooling is performed column by column along the length H to generate vertical features The formula is as follows:

[0079]

[0080] Among them, w represents the width position index of the feature map, and its value range is 0≤w≤W, j represents the height position index of the feature map, and its value range is 0≤j≤H, x c (j,w) represents the eigenvalue of channel c at position (j,w), that is, the value of the feature map at height j, width w, and channel c.

[0081] Then, a set of depth-wise separable convolutions including depth-wise convolution DWConv and point-wise convolution PWConv are used to further extract features, and then group normalization Gn is used to enhance position information. Finally, the vertical spatial attention weight y is obtained through the Sigmoid activation function σ. w The formula is as follows:

[0082]

[0083] Finally, the channel attention mechanism path and the horizontal direction path are treated as residual structures and added to the obtained output to obtain the output feature map F out :

[0084] F out =s+X c ×y h +X c ×y w ;

[0085] Therefore, by introducing the SELA-S attention mechanism module, this embodiment can improve the model's sensitivity to defect features, enhance the model's attention to defect areas, reduce the interference of background information, help the model identify defect areas more accurately, and improve detection accuracy.

[0086] At the same time, the neck network of the detection model of this embodiment includes a twelve-layer module structure connected in series, wherein the first layer is the first upsampling module, the second layer is the first Fafusion module, the third layer is the first C3K2 module, the fourth layer is the second upsampling module, the fifth layer is the second Fafusion module, the sixth layer is the second C3K2 module, the seventh layer is the first Conv module, the eighth layer is the first Concat module, the ninth layer is the third C3K2 module, the tenth layer is the second Conv module, the eleventh layer is the second Concat module, and the twelfth layer is the fourth C3K2 module.

[0087] The input end of the first Fafusion module is also connected to the second, third, and fourth C3K2 modules of the backbone network. The input end of the second Fafusion module is also connected to the first, second, and third C3K2 modules of the backbone network. The output end of the second C3K2 module is also connected to the detection head. The input end of the first Concat module is also connected to the first C3K2 module of the neck network. The output end of the third C3K2 module is also connected to the detection head. The input end of the second Concat module is also connected to the C2PSA module of the backbone network.

[0088] The neck network of this embodiment introduces two feature fusion Fafusion modules. The two Fafusion modules are connected in series, wherein the first Fafusion module is arranged after the first upsampling module of the neck network, and the output is processed by the first C3K2 module and the second upsampling module of the neck network as the input of the second Fafusion module. Both Fafusion modules include four pathways for processing feature maps, and the processing results of the four pathways are spliced ​​and the channels are adjusted as output. The four pathways include: a large-size feature map downsampling pathway, including DepthwiseConv and PointwiseConv connected in series. A medium-size feature map channel number adjustment pathway, including a Conv module for adjusting the number of feature map channels. The two small-size feature map pathways each include an Unsample module. Finally, the output images of the four pathways are concat, and then passed through the Conv adjustment channel to obtain the output feature map.

[0089] The reason why the multi-scale feature fusion pyramid network of the neck structure of this embodiment introduces the feature fusion module Fafusion on the basis of the existing neck structure is that the structural feature fusion of the existing Yolov11 relies on simple upsampling and splicing to integrate multi-level feature information, but this method is limited to continuous levels, thereby ignoring the fusion between other deep or shallow features. This local feature fusion method limits the global information sharing between features of different scales, resulting in limitations in the fusion of low-level fine-grained information and high-level semantic information. Moreover, the feature fusion between continuous levels often undergoes some other module processing, such as the C3k2 module and the CBS module in Yolov11. This feature fusion method may cause a fault in the information transmission process, which weakens the key feature information held by the fused feature level. Therefore, in order to solve these problems, this embodiment provides a multi-scale feature fusion pyramid network, which fuses multi-scale feature levels. In the first feature fusion module Fafusion, the shallow features in the yolov11 backbone network, including the third C3K2 layer and the fourth C3K2 layer, and the deep features, including the sixth C3K2 layer, are fused with the upper module of Fafusion. In the second feature fusion module Fafusion, the shallow features in the yolov11 backbone network, including the fourth C3K2 layer and the sixth C3K2 layer, and the deep features, including the eighth C3K2 layer, are fused with the upper module of Fafusion. This enhances the network's information interaction capability at different scale feature levels and ensures global information sharing between different scale features of the network.

[0090] FaFusion processes feature maps from different levels. The large-size feature map P3 undergoes depthwise separable convolution to adjust the number of feature map channels and feature map size. P4 undergoes 1×1 standard convolution to adjust the number of feature map channels. The P5 and L1 layers, which have the same number of feature map channels but smaller sizes, undergo upsampling operations to adjust the feature map size. Finally, the processed feature maps from the four different levels are concatenated and the channels are adjusted through 1×1 standard convolution to obtain the output feature map. By fusing multiple layers of features from different feature levels, the FaFusion module can effectively integrate key features from different layers of the network, enhancing the interactive fusion of low-level fine-grained information and high-level semantic information, thereby improving the expressive power of multi-scale features. This enables the model to better recognize targets of different categories in complex scenarios, further improving detection accuracy and generalization performance.

[0091] For the large-size feature map F1, there is

[0092] F1 = PWConv(DWConv(P3));

[0093] Where DWConv is depth convolution and PWConv is point convolution

[0094] For the medium-sized feature map F2, there is

[0095] F2 = Conv(P4);

[0096] For the small-size feature map F3, there is

[0097] F3 = Upsample(P5);

[0098] Upsample is upsampling

[0099] For the small-size feature map F4, there is

[0100] F4 = Upsample(L1);

[0101] Finally, F1 is concat-operated with F2, F3, and F4, and the number of channels is adjusted through Conv, so the output feature map F is obtained. out , then

[0102] F out =Conv((Concat(F1, F2, F3, F4)).

[0103] After processing the surface image, the detection model of this embodiment can output an image with the possible defect classification category, bounding box coordinates and confidence level of the defect target marked.

[0104] When training the detection model of this embodiment, the GC10-DET dataset was used to construct a related dataset. This dataset contains 10 typical steel surface defects: punched holes (Pu), welds (Wl), crescents (Cg), water spots (Water Spot), oil spots (Os), silk spots (Ss), inclusions (In), rolling pits (Rp), creases (Cr), and waist creases (Wf). Because the original dataset also includes unlabeled data and some data with incorrect labels, this paper first removes the unlabeled data from the dataset, then corrects the labels of the incorrectly labeled data. Finally, data augmentation methods including flipping, rotation, cropping, and brightness adjustment are performed. The final expanded dataset includes 4588 images.

[0105] The dataset is then divided into training set, test set, and validation set in a ratio of 8:1:1, and the training set is input into the untrained detection model for 200 rounds of iterative training. The validation set is input during the training process to obtain the verification effect and adjust the model training strategy. Finally, the training effect is verified through the test set, and the performance of the model is evaluated, thereby obtaining a trained multi-scale feature interaction steel surface defect small target detection model.

[0106] During the iterative training process of this embodiment, the backbone network extracts multi-scale features, the neck network then implements cross-layer feature interaction and fusion, and the detection head finally outputs the classification category, bounding box coordinates, and target presence confidence as the result. This result is then compared with the label value, and backpropagation is performed to update the detection model parameters. Simultaneously, a validation set is regularly fed into the detection model for verification, and the detection model parameters are further adjusted based on the verification results. Finally, after verifying the training results and evaluating the detection model's performance using a test set, the trained detection model parameters are saved.

[0107] When executing training, first divide the dataset into training, validation, and test sets in an 8:1:1 ratio. A configuration file (yaml) is created for the dataset, specifying various parameters for the dataset, including the locations of the training, validation, and test sets, the names of the categories, and the number of categories. The default.yaml file is then adjusted to adjust hyperparameters such as the number of training iterations and the choice of optimizer. The experimental environment for this example uses a 15vCPU Intel(R) Xeon(R) Platinum 8474C CPU, a 1 RTX 4090D (24GB) GPU, Python version 3.8 (Ubuntu 20.04), the deep learning framework PyTorch 1.11.0, and Cuda version 11.3.

[0108] The parameters set are as follows: the initial learning rate is set to 0.01, the momentum decay parameter value is set to 0.937, the weight decay parameter value is set to 0.000 5, the number of training rounds is set to 200, the batch size is set to 32, and the stochastic gradient descent (SGD) optimizer is used for end-to-end training. The resolution of the input image is 640×640 after preprocessing. Other parameters are set to the default values. After the setting is completed, the training set and the validation set are input into the multi-scale feature interaction steel surface defect small target detection model for 200 rounds of iterative training. The images with a resolution of 640×640 are divided into each batch for dynamic data enhancement. The first 190 rounds are spliced ​​with four-image mosaics. The enhanced batch of data will be passed through the multi-scale feature interaction model. The backbone network of the steel surface defect small target detection model with interactive multi-scale features extracts multi-scale features, and then realizes cross-layer feature interactive fusion through the multi-scale feature fusion pyramid neck network. Finally, the decoupled detection head outputs the classification category, bounding box coordinates and target existence confidence. The results are compared with the label values, and the model parameters are updated through back propagation. The model can also be verified based on the validation set. According to the verification results, some hyperparameters are adjusted to improve the network training effect. Finally, a trained multi-scale feature interaction steel surface defect small target detection model is obtained, and the optimal weight file best.pt is saved.

[0109] In the specific implementation, a high-resolution linear array CCD camera is deployed on the industrial steel continuous rolling production line to collect steel plate surface images in real time. The images are subjected to pixel normalization (mean variance normalization), Gaussian noise suppression and non-uniform illumination compensation by the preprocessing module, and then input into the trained multi-scale feature interaction steel surface defect small target detection model to obtain a model-processed image, that is, the possible defect classification category, bounding box coordinates and confidence level of the defect target are marked on the steel surface image.

[0110] In order to verify the effectiveness of the multi-scale feature interaction steel surface defect small target detection model proposed in this example technical solution,

[0111] Under the same environment and experimental parameters, the multi-scale feature interaction steel surface defect small target detection model (technical solution of the present invention) was compared with the baseline model YOLOv11 in an experimental setting. The performance indicators include mean average precision (mAP), inference speed (FPS), and parameter count (Params).

[0112] The mean average precision (mAP) is used to measure the accuracy and comprehensiveness of detection of different categories. The formula is as follows:

[0113]

[0114]

[0115] Among them, AP represents the accuracy of a single category, p(r) represents the corresponding precision under a specific recall rate r, and mAP is the result obtained by integrating the area under the PR curve. i represents the accuracy of the i-th category, represents the mean accuracy of all categories, and n is the number of categories.

[0116] FPS represents the inference speed of the mode, where t1 represents the image preprocessing time, t2 represents the image inference time, and t3 represents the image post-processing time.

[0117] FPS = 1000ms / (t1+t2+t3);

[0118] Parameters refer to the parameters that need to be learned during network model training, which represents the spatial complexity of the model. The larger the parameters, the more memory the model occupies.

[0119] The experimental results are shown in Table 1, where the mean average precision (APM) is calculated with the intersection over union (IoU) threshold set to 50%, or 0.5. The results are expressed as a percentage. Compared to the baseline model YOLOv11, this example improves the APM by 4.7% with a slight increase in parameters and a slight decrease in FPS. This significantly improves the model's accuracy in steel defect detection while maintaining minimal impact on deployability and real-time performance.

[0120] Table 1 Algorithm comparison experimental results

[0121]

[0122] Example 2:

[0123] A detection system for small target detection of steel surface defects based on a multi-scale feature interaction model, comprising:

[0124] Preprocessing module: preprocesses the acquired steel surface defect images, that is, performs data enhancement operations on the data set including flipping, rotation, cropping, and brightness adjustment data enhancement methods;

[0125] Model building module: By introducing the multi-scale feature fusion pyramid network designed in this paper into the YOLOv11 network, and introducing the feature fusion module Fafusion, the multi-scale feature extraction module SMK module, the attention mechanism module SELA-S, and the bounding box regression loss function Focaler-PIoUv2, a multi-scale feature interaction steel surface defect small target detection model is obtained;

[0126] The multi-scale feature interaction steel surface defect small target detection model consists of a backbone network, a neck network, and a detection head connected in series. The backbone network introduces a multi-scale feature extraction module (SMK module), and the output of the C2PSA module is connected to the input of the SELA-S attention mechanism module. The neck structure—a multi-scale feature fusion pyramid network—introduces a feature fusion module (Fafusion) based on the existing neck network, fusing feature maps from different levels in the backbone network. The bounding box regression loss function is Focaler-PIoUv2.

[0127] Training module: The training set obtained by preprocessing the preprocessing module is input into the multi-scale feature interaction steel surface defect small target detection model for training. During the training process, the validation set is input to adjust the model training strategy based on the validation results. Finally, the training effect is verified through the test set to evaluate the model performance, thereby obtaining a trained multi-scale feature interaction steel surface defect small target detection model.

[0128] Detection module: Use cameras to capture steel images during industrial steel production, input the trained multi-scale feature interaction steel surface defect small target detection model, and obtain the category and location information of steel surface defects as well as the result confidence.

[0129] The backbone network consists of a Conv module, a Conv module, a C3K2 module, a Conv module, a C3K2 module, a Conv module, a C3K2 module, an SMK module, a C3K2 module, an SPPF module, a C2PSA module, and a SELA-S attention mechanism module connected in series.

[0130] The Conv module consists of a standard two-dimensional convolutional layer, a BatchNormalization layer, and an activation function SILU layer connected in series.

[0131] Among them, the C3K2 module includes a Conv module and a C3K module, and the C2PSA module includes a series-connected Conv module and a PSA module.

[0132] Among them, SMK module, specifically Figure 3As shown in the figure, it includes three main pathways, namely the residual structure pathway, the local information extraction pathway and the spatial information extraction pathway. The local information pathway includes the Conv modules and the channel attention mechanism modules connected in series. The spatial information pathway includes two spatial information extraction pathways of different scales. One includes the 1×1Conv module and 3×3Conv connected in series, as well as the Dilation=2 dilated convolution module in the x-direction and the Dilation=2 dilated convolution module in the y-direction. The other includes the 1×1Conv module and 5×5Conv connected in series, as well as the Dilation=3 dilated convolution module in the x-direction and the Dilation=3 dilated convolution module in the y-direction. Finally, the feature map processed by the residual structure pathway is concat with the feature map processed by the local information extraction pathway and the spatial information extraction pathway to obtain the final output feature map.

[0133] Among them, the SELA-S attention mechanism module is as follows Figure 4 As shown in the figure, it includes a spatial attention mechanism pathway and a channel attention mechanism pathway. The spatial attention mechanism includes two average pooling layers and feature extraction layers in two different directions, vertical and horizontal, to obtain spatial feature weights in two different directions. The channel attention mechanism is an adaptive pooling layer and a convolution layer connected in series, and finally passes through a sigmoid function to obtain the channel feature weight. Finally, each weight information is multiplied with the original feature map, and added to the original feature map as the residual branch to obtain the output feature map.

[0134] Among them, the neck structure multi-scale feature fusion pyramid network adjusted the neck fusion route based on the existing neck structure and introduced the feature fusion module Fafusion. Through the fusion of multi-scale feature levels, the network fused the shallow features of the yolov11 backbone network including the third C3K2 layer and the fourth C3K2 layer, the deep features of the sixth C3K2 layer, and the upper modules of the first feature fusion module Fafusion in the first feature fusion module Fafusion. In the second feature fusion module Fafusion, the shallow features of the yolov11 backbone network including the fourth C3K2 layer and the sixth C3K2 layer, the deep features of the eighth C3K2 layer, and the upper modules of the second feature fusion module Fafusion were fused.

[0135] Furthermore, the feature fusion module Fafusion is as follows Figure 4As shown in the figure, it includes four pathways. The first is the large-size feature map downsampling pathway, which includes DepthwiseConv and PointwiseConv connected in series. The second is the medium-size feature map channel adjustment pathway, which includes a Conv module for adjusting the number of feature map channels. The third and fourth pathways include two small-size feature map pathways respectively. Both pathways include an Unsample module. Finally, the output images of the four pathways are concat- ed and then pass through the Conv adjustment channel to obtain the output feature map.

[0136] Among them, Focaler-PIoUv2 is a bounding box loss function, Focaler-IoU is a bounding box loss function designed to enable the model to focus on different regression samples in different detection tasks, and PIoUv2 is a bounding box loss function that constructs a bounding box regression loss function by quantitatively minimizing the Euclidean distance between the predicted box and the true box, thereby enhancing the bounding box regression. This method provides a more intuitive similarity measure and can guide the anchor box to regress along an effective path, solving the problem of slow convergence caused by the increase in the traditional IoU anchor box, and ensuring accurate and stable bounding box regression. The two are combined into the Focaler-PIoUv2 bounding box loss function, which provides a more effective bounding box regression loss function for the multi-scale feature interaction steel surface defect small target detection model, enabling the model to focus on different regression samples in different detection tasks, while ensuring more accurate and stable bounding box regression.

[0137] It should be understood that the specific implementation process of each module please refer to the above method content, the present invention will not go into details here, and the division of the above functional modules is only for example illustration. In some embodiments, some functional modules can be merged, and some functional modules can be split. Each functional module can be implemented in software or hardware or a combination of software and hardware. Among them, the software and hardware equipment includes but is not limited to general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0138] Example 3:

[0139] A computer-readable storage medium stores a computer program that is called by a processor to implement:

[0140] The steps of the above-mentioned steel surface defect detection method.

[0141] For the specific implementation process of each step, please refer to the description of the above method.

[0142] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0143] Example 4:

[0144] The present invention provides a computer-readable storage medium storing a computer program, which is called by a processor to implement the steps of the above-mentioned steel surface defect detection method.

[0145] For the specific implementation process of each step, please refer to the description of the above method.

[0146] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the software and hardware device of any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard disk equipped on the controller, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Furthermore, the readable storage medium can also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store computer programs and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0147] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned readable storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0148] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is a flow chart according to the method, device (system), and computer program product of the embodiment of the present application and / or the instructions executed by the processor to generate a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product comprising an instruction device, which realizes the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0149] It should be emphasized that the examples of the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A method for detecting small targets of surface defects based on multi-scale feature interaction, characterized in that: The following steps are involved: Step 1: Input the surface image of the product to be inspected, which is collected in real time, into the inspection model; Step 2: After the detection model processes the surface image, it outputs an image with the possible defect classification category, bounding box coordinates, and confidence level of the defect target; The detection model is based on the YOLOv11 network as the basic structure, including the backbone network, neck network and detection head connected in series. The backbone network introduces a multi-scale feature extraction SMK module and an attention mechanism SELA-S module; the neck network introduces two feature fusion Fafusion modules; The SMK module is set between the third and fourth C3K2 modules of the backbone network, and includes a residual structure path, a local information extraction path, and a spatial information extraction path for processing the feature map input to the SMK module respectively. The processing results of the three paths are spliced ​​as output; The SELA-S module is set between the original SPPF module and the C2PSA module of the backbone network, and includes a spatial attention mechanism pathway and a channel attention mechanism pathway for processing the feature maps input to the SELA-S module respectively. The processing results of the two pathways are multiplied by the input feature maps respectively, and the multiplication results are added to the input feature maps as output; The two Fafusion modules are connected in series, wherein the first Fafusion module is arranged after the first upsampling module of the neck network, and the output is processed by the first C3K2 module and the second upsampling module of the neck network as the input of the second Fafusion module; both Fafusion modules include four paths for processing feature maps, and the processing results of the four paths are spliced ​​and the channels are adjusted as output.

2. The method according to claim 1, characterized in that In step 1, the detection model is trained by the following steps: Step 101: Obtain surface images of products with surface defects, annotate them and use them as a dataset, then perform data augmentation on the images in the dataset to expand the dataset. In step 102, the expanded data set is divided into a training set, a validation set, and a test set according to a preset ratio. The training set is then input into the detection model for iterative training. During the iterative training process, the validation set is periodically input into the detection model to adjust the model training strategy according to the verification effect. Finally, after the iterative training is completed, the training effect is verified and the performance of the detection model is evaluated by inputting the test set.

3. The method according to claim 2, characterized in that In step 101, the data enhancement is to perform processing including flipping, rotating, cropping and brightness adjustment on the image, thereby generating a processed new image based on the original image to expand the data set.

4. The method according to claim 2, characterized in that In step 102, during the iterative training process, multi-scale features are extracted through the backbone network, and then the neck network realizes cross-layer feature interaction and fusion. Finally, the detection head outputs the classification category, bounding box coordinates and target existence confidence as the result; the result is then compared with the label value, and back propagation is performed to update the parameters of the detection model; at the same time, a validation set is regularly input into the detection model for verification, so as to further adjust the parameters of the detection model according to the verification results; finally, after verifying the training effect and evaluating the performance of the detection model through the test set, the parameters of the detection model obtained by training are saved.

5. The method according to claim 1, wherein The backbone network of the detection model includes a twelve-layer module structure connected in series, wherein the first layer is the first Conv module, the second layer is the second Conv module, the third layer is the first C3K2 module, the fourth layer is the third Conv module, the fifth layer is the second C3K2 module, the sixth layer is the fourth Conv module, the seventh layer is the third C3K2 module, the eighth layer is the SMK module, the ninth layer is the fourth C3K2 module, the tenth layer is the SPPF module, the eleventh layer is the SELA-S attention mechanism module, and the twelfth layer is the C2PSA module.

6. The method according to claim 5, characterized in that The SMK module includes: The residual structure path includes a 1×1 Conv module; The local information extraction pathway includes a Conv module and a channel attention mechanism module connected in series; The spatial information extraction pathway includes two sub-pathways of different scales. One of them includes a 1×1 Conv module, a 3×3 Conv module, a dilated convolution module with a dilation of 2 in the x-direction, and a dilated convolution module with a dilation of 2 in the y-direction, which are connected in series. The other sub-pathway includes a 1×1 Conv module, a 5×5 Conv module, a dilated convolution module with a dilation of 3 in the x-direction, and a dilated convolution module with a dilation of 3 in the y-direction, which are connected in series. Concat module for concatenating the residual structure pathway, local information extraction pathway, and spatial information extraction pathway.

7. The method according to claim 5, characterized in that The SELA-S attention mechanism module includes: The spatial attention mechanism pathway includes two sets of spatial feature weight extraction modules connected in series. Each set of modules includes a connected average pooling layer, a feature extraction layer, group normalization, and a sigmoid function. The average pooling layer of one set of modules is a vertical average pooling layer, and the average pooling layer of the other set is a horizontal average pooling layer. The output of the first set of modules is multiplied by the feature map of the input SELA-S attention mechanism module and then input into the second set of modules to obtain spatial feature weights in two different directions. The channel attention mechanism pathway includes a series of adaptive pooling layers and convolutional layers, and finally passes through a sigmoid function to obtain the channel feature weights. The processing results of the two pathways, including the spatial feature weights in the vertical and horizontal directions, and the channel feature weights obtained by the channel attention mechanism pathway, are then multiplied with the original feature map respectively. Finally, each result is added to the original feature map as the residual branch as the output.

8. The method according to claim 5, characterized in that The neck network of the detection model includes a twelve-layer module structure connected in series, wherein the first layer is a first upsampling module, the second layer is a first Fafusion module, the third layer is a first C3K2 module, the fourth layer is a second upsampling module, the fifth layer is a second Fafusion module, the sixth layer is a second C3K2 module, the seventh layer is a first Conv module, the eighth layer is a first Concat module, the ninth layer is a third C3K2 module, the tenth layer is a second Conv module, the eleventh layer is a second Concat module, and the twelfth layer is a fourth C3K2 module; The input end of the first Fafusion module is also connected to the second C3K2 module, the third C3K2 module and the fourth C3K2 module of the backbone network respectively; the input end of the second Fafusion module is also connected to the first C3K2 module, the second C3K2 module and the third C3K2 module of the backbone network respectively; the output end of the second C3K2 module is also connected to the detection head; the input end of the first Concat module is also connected to the first C3K2 module of the neck network; the output end of the third C3K2 module is also connected to the detection head; the input end of the second Concat module is also connected to the C2PSA module of the backbone network.

9. The method according to claim 1, characterized in that The Fafusion module in the neck network has four pathways: a large-size feature map downsampling pathway, comprising DepthwiseConv and PointwiseConv connected in series; a medium-size feature map channel number adjustment pathway, comprising a Conv module for adjusting the number of feature map channels; and two small-size feature map pathways, each comprising an Unsample module. Finally, the output images of the four pathways are concat- ed and then passed through the Conv adjustment pathway to obtain the output feature map.

Images