Steel surface defect detection method based on edge enhancement and double-flow fusion

Abstract

A method for detecting surface defects in steel based on edge enhancement and dual-stream fusion, relating to the fields of computer vision and artificial intelligence, includes the following steps: Step S1, unifying the framework of the backbone network of the LEGNet neural network model; Step S2, performing overflow prevention correction on the Gaussian-Laplacian kernel operator and Scharr operator in the BasicStage module of LEGNet; Step S3, extracting the BasicStage module from LEGNet, and combining the BasicStage module with downsampling, normalization layers, and activation functions to form a neural network structure, thus constructing an ERA module, etc. This invention utilizes the LEGNet Gaussian-Laplacian kernel module, which can effectively suppress noise that damages edge and texture features, thereby improving the detection accuracy of the model to a certain extent.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method for detecting surface defects in steel based on edge enhancement and dual-stream fusion. Background Technology

[0002] With the widespread application of deep learning in computer vision, object detection algorithms have become a core technology in fields such as intelligent manufacturing. These algorithms are primarily used for detecting surface defects in industrial products. Existing mainstream detection networks such as the YOLO series, Faster R-CNN, and SSD have achieved significant results in conventional object detection. However, in industrial surface defect detection, especially for steel surface data, the morphology of defects is diverse. Referring to the NEU defect detection dataset, defects can be categorized into six typical types: cracks, inclusions, patches, pitting corrosion, rolled oxide scale, and scratches. Detection of these defects is challenging, and current defect detection models suffer from the following problems:

[0003] (1) Insufficient feature fusion capability. Due to the black box nature of the model, the global feature fusion of the traditional feature pyramid structure (FPN) in the target detection model is not very targeted in feature enhancement. Although some studies have proposed dual backbone networks for feature fusion, unreasonable dual network structure design will lead to increased computation and may also cause different features to interfere with each other.

[0004] (2) Insufficient utilization of edge information. Industrial defects often manifest as brightness perturbations, geometric deformations or texture anomalies, among which edge features are the most significant. However, ordinary convolutional networks are not sensitive to edge gradients, which can easily lead to blurred boundaries. At the same time, edge features are highly sensitive to noise, and networks with weak noise resistance have difficulty effectively processing noise.

[0005] (3) Insufficient global context modeling capability. Industrial defects often have strong correlation with image context information, and the image information around the target area is also an indispensable condition for successful detection. However, networks that rely solely on local convolution operations are unable to capture global dependencies and have a high false detection rate in complex backgrounds.

[0006] (4) Imbalance between the number of detection algorithm parameters and the performance of the detection device. Industrial defect detection requires algorithms to be real-time, lightweight, and easy to deploy. However, lightweight and fast network models often do not have high detection accuracy. In contrast, strong detection models, such as almost all two-stage target detection networks, often have a large number of parameters and slow response speed. At the same time, the use of different deep learning libraries in the design and development of many algorithm networks is not conducive to the unification of detection algorithms, which in turn increases the difficulty of deployment.

[0007] (5) Mismatch between pre-trained models and target tasks. Due to the confidentiality principles of enterprise technology, there are relatively few datasets available for training defect detection models, and some datasets are of low quality. Furthermore, the traditional method of using ImageNet-based pre-trained models for defect detection tasks does not effectively transfer the learned feature distribution to the image edge and texture defect detection tasks that require processing. Forced transfer sometimes actually reduces the model's detection accuracy. At the same time, pre-training often involves significant time consumption, and the resulting weights occupy a large amount of storage space, making pre-training a problem that needs to be addressed in defect detection tasks.

[0008] In the field of remote sensing image detection, which also requires edge enhancement, edge enhancement algorithms are often integrated into network models, achieving quite good results in some object detection tasks. In recent years, the Transformer architecture has been introduced into machine vision and has matured through long-term research. Typical representative models include VisionTransformer and Swing Transformer. This series of models improves context modeling through a global self-attention mechanism, performing well in many detection tasks. Further improvements by introducing attention models, such as the YOLO series, have demonstrated high accuracy. However, these models still have room for improvement due to slow convergence speed and difficulty in adapting to different target tasks. Dual-backbone networks like YOLOFuse attempt to extract semantic and structural features through two parallel branches, but feature fusion efficiency and edge feature representation remain limited. Furthermore, the models have a large number of parameters and high computational cost, requiring further improvements to move from theory to practical application. Summary of the Invention

[0009] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a defect and small target detection method and system based on edge feature enhancement and dual backbone fusion, which solves the technical problems of insufficient feature fusion capability, insufficient utilization of edge information, and insufficient global context modeling capability.

[0010] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0011] A method for detecting surface defects in steel based on edge enhancement and dual-flow fusion includes the following steps:

[0012] Step S1: Unify the framework of the backbone network of the LEGNet neural network model using PyTorch, and configure the backbone network using the Ultralytics framework.

[0013] Step S2: Perform overflow prevention correction for the Gaussian-Laplacian kernel and Scharr operator in the BasicStage module of LEGNet, including adding small offsets to the convolution kernel K and Scharr operator in the Gaussian-Laplacian kernel. ;

[0014] Step S3: Extract the BasicStage module from LEGNet, combine the BasicStage module with downsampling, normalization layers and activation functions to form a neural network structure, and provide function selection functionality to form the ERA module;

[0015] Step S4: Connect the BasicStage module that constitutes the ERA module to the Swin Transformer window attention mechanism to construct the Gaussian-Edge Attention Mechanism (EGA).

[0016] Step S5: Gated or residual fusion is performed between the backbone features F extracted from the backbone network and the F processed by the ERA module, using learnable coefficients. By controlling the enhancement intensity, a detection model is obtained;

[0017] Step S6: Prepare the NEU defect detection dataset. The NEU defect detection dataset is a dataset of steel surface defect images used for detection. Read the images from the NEU defect detection dataset, perform preprocessing and image enhancement operations, and then input them into the detection model to obtain the detection results.

[0018] Furthermore, in step S1, configuring the backbone network using the Ultralytics framework includes integrating the modules required by the detection model into the module library under Ultralytics, and splicing them into a complete network structure in a unified format.

[0019] Furthermore, in step S2, a small offset is added to the convolution kernel K in the Gaussian-Laplacian kernel operator. The operator is then modified as follows:

[0020] ;

[0021] Where x and y represent pixel coordinates, a tiny offset. .

[0022] Furthermore, in step S2, the Scharr operator adds a small offset. The operator is then modified as follows:

[0023] ;

[0024] Where M represents the synthetic edge intensity, F x Fy Represent , I represents the feature map matrix. Represents the edge extraction factor in the x and y directions. For convolution, a small offset .

[0025] Furthermore, step S3 also includes upsampling or downsampling before feature fusion to unify the spatial resolution of the image.

[0026] Furthermore, in step S4, the BasicStage module constituting the ERA module is connected to the SwinTransformer window attention mechanism, including a shallow layer with an attention block guided by EGA.

[0027] Furthermore, in step S5, the learnable coefficients are used... The enhancement strength is controlled, including: when using gating mode, the fusion form is as follows:

[0028] ;

[0029] Among them, F out F represents the output feature, G(•) represents the structural input feature, and G(•) represents the gated graph generated by convolution and the sigmoid function. leg The backbone features are processed by the ERA module and are used to adaptively adjust the edge enhancement level in both spatial and channel dimensions.

[0030] Furthermore, in step S5, the learnable coefficients are used... Controlling the enhancement intensity includes: when using the residual mode, the fusion form is as follows:

[0031] ;

[0032] Among them, F out F represents the output feature, F represents the structural input feature, and G(•) represents the gated graph generated by convolution and the sigmoid function. leg These are the backbone features after processing by the ERA module.

[0033] Furthermore, in step S6, the NEU defect detection dataset is augmented offline to a size 10 times larger than the original dataset.

[0034] Furthermore, in step S6, the preprocessing operations include scaling and normalization; the image enhancement operations include random rotation, mosaic enhancement, and MixUp.

[0035] The beneficial effects of this invention are as follows: This invention can enhance the edge of the feature map of the deep network by localization fusion. By using the LEGNet Gaussian-Laplacian kernel module, it can effectively suppress noise that destroys features such as edges and textures, thereby improving the detection accuracy of the model to a certain extent. By introducing the Swing Transformer sliding window attention mechanism, it can realize the association of image patch information and feature optimization, thereby effectively utilizing contextual information to solve the detection problem.

[0036] This invention not only balances edge detail preservation and global semantic modeling, but also provides a novel detection structure that can be trained and completed using existing defect detection datasets. It improves the performance of small target detection while maintaining lightweight design, thereby enhancing accuracy. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the process for detecting surface defects in steel using the method of the present invention;

[0038] Figure 2 This is a schematic diagram of the ERA edge enhancement module processing flow;

[0039] Figure 3 This is a schematic diagram of the processing flow of the BasicStage module;

[0040] Figure 4 for Figure 3 The flowchart of "stacked Swing Transformer attention blocks x depth" is shown in the figure. Detailed Implementation

[0041] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] This invention provides a method for detecting surface defects in steel based on edge enhancement and dual-flow fusion, comprising the following steps:

[0043] Step S1: Use PyTorch to perform unified framework operations on the backbone network of the LEGNet neural network model, and use the Ultralytics framework to configure the backbone network.

[0044] This invention is based on the Ultralytics framework and uses a pure PyTorch library to build a deep learning environment. LEGNet, proposed by Wei Lu et al. in 2025, is a network model for rotating target detection in remote sensing images. However, it is implemented using PyTorch, MMCV, MMDetection, and the timm library. While MMCV and MMDetection are powerful deep learning frameworks, they suffer from difficulties in construction, stringent configuration requirements, high learning costs, and are unsuitable for deployment on edge devices. PyTorch, on the other hand, is a widely used deep learning library with high industry recognition, and its lightweight modifications can adapt it to more devices with limited performance. Although the timm library provides modules for batch normalization and dropout, it has strict requirements for multiple libraries, such as NumPy and PyTorch versions, making it prone to deployment failures, difficult to maintain, and lacking in portability. Therefore, this invention comprehensively rewrites the key part of LEGNet, namely the backbone network, using PyTorch. This includes changing the batch normalization operation from MMDetection support to pure PyTorch support, and changing the DropPath mechanism from MMDetection format to PyTorch format, thus effectively solving the incompatibility issues between MMDetection and MMCV and other libraries. Simultaneously, to facilitate network code configuration and maintenance, this invention configures the backbone network using the Ultralytics framework, similar to the YOLO series models. The modules required by the detection model of this invention are integrated into the module library under Ultralytics, and assembled into a complete network structure in a unified format. Subsequent training experiments using the NEU steel defect detection dataset demonstrate the rationality and usability of the modifications.

[0045] Step S2: Perform overflow prevention correction for the Gaussian-Laplacian kernel and Scharr operator in the BasicStage module of LEGNet, including adding small offsets to the convolution kernel K and Scharr operator in the Gaussian-Laplacian kernel. .

[0046] The key component for implementing this step is the Edge Refine Adapter (ERA), which is inspired by the open-source LEGNet neural network model. This lightweight edge enhancement network offers some decomposability and portability. However, experiments on LEGNet's defect detection reveal significant numerical computational issues. Another important source of the ERA module is the open-source dual-backbone neural network YOLOFuse. This structure directly designs two originally independent deep networks in parallel, achieving feature fusion at the head layer through early, mid, or decision-level fusion to improve detection performance. Theoretically, this structure can achieve feature fusion and improve detection performance; however, after training, the network will have two training weights simultaneously, which will significantly consume the storage space of edge devices. Furthermore, the layer output features of the two networks may not be spatially aligned, and forced mixing may lead to mutual interference between features, ultimately lowering the overall accuracy.

[0047] This invention provides an "overflow prevention correction" for the Gaussian-Laplacian kernel operator in LEGNet. The LEGNet source code uses the Gaussian-Laplacian operator and the Scharr operator for image edge feature extraction and noise suppression. In the original Gaussian-Laplacian kernel operator, the normalization operation for the convolution kernel K is as follows:

[0048] ;

[0049] In the formula, x and y represent pixel coordinates. This operation exists mathematically. The case of being equal to zero or extremely small leads to If the value of the operator is missing or overflows, its generalization performance will be compromised in the field of image processing. In experiments, it's common to see normal accuracy metrics like mAP50 during training, but all accuracy-related metrics become 0 during validation. On some datasets, the training set loss may be NaN (a non-data type, often representing infinity). Therefore, this invention addresses this problem by modifying the operator as follows:

[0050] ;

[0051] In the formula, x and y represent pixel coordinates, and the offset is small. By adding a tiny offset The operator can be modified.

[0052] In this case, the original formula for the Scharr operator is:

[0053] ;

[0054] Where M represents the synthetic edge intensity, used to measure the strength of edge features; F x F y Represent , I represents the feature map matrix. Represents the edge extraction factor in the x and y directions. For convolution, a small offset .

[0055] However, when the gradient of the input feature is close to zero in the region, the value within the square root in the above formula may be extremely small or even zero, causing instability in the calculation process at floating-point precision. Therefore, this invention provides an overflow prevention correction:

[0056] ;

[0057] Among them, tiny offset With the above modifications, the revised LEGNet can be trained and validated normally on both the COCO-2017 dataset and the NEU steel defect detection dataset.

[0058] Step S3: Extract the BasicStage module from LEGNet, and combine the BasicStage module with downsampling, normalization layers, and activation functions to form a neural network structure, providing a function selection function to constitute the ERA module. Specifically, step S3 also includes using upsampling or downsampling before feature fusion to unify the spatial resolution of the image.

[0059] This invention deconstructs and reconstructs the LEGNet structure. While retaining the edge enhancement concept of the LEGNet network, it modularizes its structure to achieve dual-stream collaboration with various backbone networks. Through modularization, LEGNet is reconstructed into an embeddable, lightweight ERA module. Specifically, the BasicStage module of LEGNet is extracted. This is one of the core modules responsible for edge feature enhancement in the network. It incorporates the Scharr edge enhancement operator, the Gaussian smoothing operator, and well-known operators in image processing. Its formula is:

[0060] ;

[0061] in: The midpoint of the Gaussian kernel The weights; y represents the standard deviation, used to control the degree of blur; x and y are coordinates relative to the kernel center. The BasicStage module, along with downsampling, normalization layers, and activation functions, forms a simple neural network structure, providing function selection capabilities. This neural network structure is the ERA module, and its image processing flow is as follows: Figure 2 As shown. ERA uses the mid-layer output features of the backbone network as input to this module, and utilizes the BasicStage module to extract and enhance edge and texture information. The BasicStage module is shown below. Figure 3 As shown, the module's parameter configuration will change depending on the stage number. In addition, the Scharr operator is configured in stage 0, and the Gaussian operator is configured in the later stages, but the basic operations are edge enhancement and noise suppression.

[0062] Step S4: Connect the BasicStage module in the ERA module to the Swin Transformer window attention mechanism to construct the Gaussian-Edge Attention Mechanism (EGA).

[0063] To enhance the feature extraction capability of the BasicStage module, this invention incorporates a Swing Transformer window attention mechanism within the BasicStage module for feature map optimization, thereby constructing an Edge-Gaussian Attention (EGA) mechanism to better enhance image edges.

[0064] The BasicStage module in the original LEGNet model is relatively simple in design and lacks sufficient detection power for some defects. This invention adds the sliding window attention mechanism from the Swing Transformer to the BasicStage module. The specific addition process is as follows: Figure 3 As shown in the figure, "STACK" represents "stacked Swing Transformer attention blocks". The repetitive simplification of the "depth" statement was demonstrated through theoretical analysis and fine-tuning experiments, proving the rationale for adding the module.

[0065] Shallow attention preserves the richest geometric details (edges, textures, corners): In model training experiments, after enabling mixed-precision training or deep downsampling, many details are "smoothed out" by convolution / pooling at deeper scales. Applying an attention block with EGA guidance to shallow layers can highlight boundaries and high-frequency textures at high resolution; in experiments, the introduction of shallow attention modules improved the model's detection accuracy for samples with scratches and defects to some extent.

[0066] "Local Adaptation" of Window Attention: Theory and experiments have shown that the "local adaptation" principle used by the window attention of Swing Transformer, compared with the global attention used by ViT (Vision Transformer), can improve the feature extraction level while ensuring that the computational complexity is not too high. In the experiments corresponding to this embodiment, it is reflected in the improvement of the accuracy of some defect detection, while also showing a smaller computational overhead.

[0067] Fixed convolution kernels have limited adaptability to different texture orientations / scales; Swing's local window plus staggered shift can provide "learnable, orientation / position adaptive" geometric modeling, while keeping the computation within the local window, with controllable complexity, making it suitable for shallow, high-resolution layers.

[0068] Suppress image noise and enhance the ERA module: such as Figure 3 As shown, adding an attention branch at the Scharr module in S0 (shallow layer) can a priori amplify edge saliency, guide attention weights to the structural boundary, and improve the consistency between the contour and the instance boundary. The specific process is as follows: Figure 4 As shown, this is reflected in the improved heat map in the experiment.

[0069] Step S5: Gated or residual fusion is performed between the backbone features F extracted from the backbone network and the F processed by the ERA module, using learnable coefficients. By controlling the enhancement intensity, a detection model is obtained.

[0070] To avoid direct conflicts between edges and semantic features, this invention designs an adaptive fusion structure based on a gated residual mechanism, such as... Figure 2 As shown. This structure is achieved through... Figure 1 The backbone features F extracted from the backbone network are gated or fused with the F processed by the ERA module, and the learnable coefficients are used to determine the final result. By controlling the enhancement intensity, a detection model is obtained.

[0071] When using the gating mode, the fusion form is as follows:

[0072] ;

[0073] Among them, F out F represents the output feature, G(•) represents the structural input feature, and G(•) represents the gated graph generated by convolution and the sigmoid function. leg The backbone features are processed by the ERA module and are used to adaptively adjust the edge enhancement level in both spatial and channel dimensions.

[0074] When using the residual mode, the fusion form is as follows:

[0075] ;

[0076] Among them, F out F represents the output feature, F represents the structural input feature, and G(•) represents the gated graph generated by convolution and the sigmoid function. leg These are the backbone features after processing by the ERA module.

[0077] Through the aforementioned adaptive fusion mechanism, the ERA module can enhance target boundaries and texture details while maintaining the stability of the backbone semantics, thus effectively enhancing edge-sensitive features.

[0078] Furthermore, the embedding of network modules in the fusion layer enables gradient sharing to a certain extent, thus ensuring the consistency of feature learning and the stability of parameter updates during backpropagation. Learnable coefficients The introduction of this feature enables the enhancement amplitude to adapt to different tasks and defect types, further improving the model's versatility and robustness.

[0079] Step S6: Prepare the NEU defect detection dataset. The NEU defect detection dataset is a dataset of steel surface defect images used for detection. The NEU defect detection dataset is expanded to 10 times the size of the original dataset through offline enhancement. Images are read from the NEU defect detection dataset, and after preprocessing operations such as scaling and normalization, as well as image enhancement operations such as random rotation, mosaic enhancement, and MixUp, they are input into the detection model to obtain the detection results.

[0080] The input image undergoes the following process in the detection model of this invention:

[0081] 1. Extract features from the read-in image in the backbone network to obtain multi-scale feature maps such as mid-level features.

[0082] 2. Construct a dual-branch system, inputting the mid-level feature C3 into two branches simultaneously. One branch uses the ERA edge enhancement module to enhance the low-level features of C3, while the other branch continues to use the network feature extraction module to transform C3 into high-level features C4.

[0083] 3. The branch containing C4 is divided into two branches, and C4 is simultaneously input into both branches. One branch uses the ERA edge enhancement module to enhance edge features, while the other branch extracts deep features C5.

[0084] 4. Multi-scale feature fusion is performed on C3, C4 and depth feature C5 from the edge enhancement of the ERA module. A total of three scale feature map pairs are spliced ​​in the image channel dimension to obtain the fused features P3, P4 and P5.

[0085] 5. Apply the detection head object detection model to P3, P4, and P5 respectively to predict and classify the detection boxes, and obtain the detection results. Here, the detection head is the last component of the object detection model, used to perform operations such as detection box prediction, object classification, and probability generation on the model's feature map, thereby generating the model's critical results for the image.

[0086] 6. By comparing the improved method with the existing method, the rationality and effectiveness of the improvement are demonstrated.

[0087] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Any modifications, alterations, substitutions, and variations made by those skilled in the art to the above embodiments are within the scope of the present invention.

Claims

1. A method for detecting surface defects in steel based on edge enhancement and dual-flow fusion, characterized in that, Includes the following steps: Step S1: Unify the framework of the backbone network of the LEGNet neural network model using PyTorch, and configure the backbone network using the Ultralytics framework. Step S2: Perform overflow prevention corrections on the Gaussian-Laplacian kernel and Scharr operator in the BasicStage module of LEGNet, including adding small offsets to the convolution kernel K and the Scharr operator in the Gaussian-Laplacian kernel. ; Step S3: Extract the BasicStage module from LEGNet, combine the BasicStage module with downsampling, normalization layers and activation functions to form a neural network structure, and provide function selection functionality to form the ERA module; Step S4: Connect the BasicStage module that constitutes the ERA module to the Swin Transformer window attention mechanism to construct the Gaussian-Edge Attention Mechanism (EGA). Step S5: Gated or residual fusion is performed between the backbone features extracted from the backbone network and the backbone features processed by the ERA module, using learnable coefficients. By controlling the enhancement intensity, a detection model is obtained; Step S6: Prepare the NEU defect detection dataset. The NEU defect detection dataset is a dataset of steel surface defect images used for detection. Read the images from the NEU defect detection dataset, perform preprocessing and image enhancement operations, and input them into the detection model to obtain the detection results, proving the rationality of the improvement.

2. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: In step S1, configuring the backbone network using the Ultralytics framework includes integrating the modules required by the detection model into the module library under Ultralytics, and splicing them into a complete network structure in a unified format.

3. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: In step S2, a small offset is added to the convolution kernel K in the Gaussian-Laplacian kernel operator. The operator is then modified as follows: ； Where x and y represent pixel coordinates, a tiny offset. .

4. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: In step S2, the Scharr operator adds a small offset. The operator is then modified as follows: ； Where M represents the synthetic edge intensity, F x F y Represent , I represents the feature map matrix. Represents the edge extraction factor in the x and y directions. For convolution, a small offset .

5. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: Step S3 further includes upsampling or downsampling before feature fusion to unify the spatial resolution of the image.

6. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: In step S4, the BasicStage module constituting the ERA module is connected to the Swin Transformer window attention mechanism, including a shallow layer with an attention block guided by EGA.

7. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: In step S5, the learnable coefficients are used. The enhancement strength is controlled, including: when using gating mode, the fusion form is as follows: ； Among them, F out F represents the output feature, G(•) represents the structural input feature, and G(•) represents the gated graph generated by convolution and the sigmoid function. leg These are the backbone features after processing by the ERA module.

8. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: In step S5, the learnable coefficients are used. Controlling the enhancement intensity includes: when using the residual mode, the fusion form is as follows: ； Among them, F out F represents the output feature, and F represents the structural input feature. leg These are the backbone features after processing by the ERA module.

9. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: In step S6, the NEU defect detection dataset is augmented offline to 10 times the size of the original dataset.

10. The method for detecting steel surface defects based on edge enhancement and dual-flow fusion according to claim 1, characterized in that: In step S6, the preprocessing operations include scaling and normalization; the image enhancement operations include random rotation, mosaic enhancement, and MixUp.

Images