A Deep Learning-Based End-to-End Detection Method and System for Steel Surface Defects

Abstract

This invention relates to the field of industrial defect processing technology, specifically disclosing an end-to-end detection method, system, and apparatus for steel surface defects based on deep learning. The system automatically acquires and preprocesses images of the steel surface through a hardware device that coordinates material transport, sensor triggering, and light source interaction. The images are then input into an improved YOLOv11 defect detection model, which uses MobileNetV4 as a lightweight backbone network and combines a BiFPN weighted bidirectional pyramid network with MCAttention multi-scale cross-axis attention mechanism to achieve efficient feature extraction and accurate localization of multi-scale, low-contrast, and subtle defects. Finally, the detection results are sent to the DeepSeek AI intelligent analysis module for in-depth analysis of defect causes and output of process optimization suggestions, automatically generating a detection report. This invention achieves an end-to-end fully automated closed-loop process from image acquisition to result diagnosis.

Description

Technical Field

[0001] This invention relates to the field of industrial defect handling technology, and more specifically, to a deep learning-based end-to-end detection method and system for steel surface defects. Background Technology

[0002] As the steel manufacturing industry develops towards continuous, intelligent, and high-quality production, surface defect detection has become a crucial link in quality control of hot-rolled, cold-rolled, strip, and plate production processes. During steel production and processing, defects such as cracks, scratches, inclusions, patches, oxide scale, pitting, and indentations may appear on the surface. These defects not only affect the appearance quality of the steel but may also impact subsequent coating, forming, welding, and service reliability. Therefore, in high-speed production line environments, how to quickly, accurately, and stably identify and locate defects in steel surface images is a critical problem that steel companies urgently need to solve in their quality inspection systems.

[0003] In existing technologies, early steel surface defect detection mainly relied on manual visual inspection or traditional machine vision-based methods. Manual visual inspection is heavily influenced by the inspector's experience, fatigue, and subjective judgment, making it difficult to meet the high-speed, high-consistency, and traceability requirements of continuous production lines. Traditional machine vision methods typically achieve defect identification through grayscale thresholding, edge detection, texture feature extraction, morphological processing, and classifier discrimination. These methods can achieve certain results under stable lighting conditions, with regular defect shapes and minimal background interference. However, in real industrial environments, steel surfaces suffer from issues such as reflections, oil stains, water stains, oxidation color differences, texture disturbances, and large differences in defect scale. This makes traditional methods highly sensitive to environmental changes, resulting in insufficient generalization ability and robustness.

[0004] To address the aforementioned issues, deep learning-based object detection methods are commonly introduced. These methods automatically extract semantic features from steel surface images using convolutional neural networks and combine them with feature pyramids, multi-scale detection heads, attention mechanisms, or lightweight network structures to achieve defect category identification and defect location regression. Existing mainstream infrastructure typically includes an image acquisition and preprocessing module, a backbone feature extraction network module, a multi-scale feature fusion network module, and a detection output module. The backbone network extracts image features at different levels, the feature pyramid network fuses shallow spatial details and deep semantic information, the attention mechanism enhances the feature response of defect-related regions, and the detection head outputs defect category, confidence level, and location information. However, in actual steel surface defect detection scenarios, defects often exhibit characteristics such as elongated shape, weak texture contrast, small target distribution, multi-scale coexistence, and significant directionality. Conventional convolutional feature extraction and ordinary multi-scale fusion methods still struggle to adequately model the long-range correlations of defects in the horizontal and vertical directions, easily leading to problems such as missed detection of small defects, false detection of similar defects, and unstable localization in complex backgrounds.

[0005] A search revealed several existing deep learning solutions for detecting surface defects in steel or strip. For example, Chinese invention patent CN112819748B, entitled "A Training Method and Apparatus for a Strip Surface Defect Recognition Model," discloses a technical solution for acquiring strip surface image samples, extracting target feature maps using a neural network model with multi-scale feature fusion capabilities, and training a strip surface defect recognition model based on the target feature maps. Its main purpose is to improve the recognition effect of strip surface defects. While this solution introduces the concept of multi-scale feature fusion, its focus is on the training process of the defect recognition model and multi-scale feature extraction. It does not establish an interactive enhancement mechanism along different spatial axes for common steel surface defects such as elongated, directional, and weak-contrast defects. Furthermore, it does not disclose the specific structure for learnable weight normalization allocation of different input feature branches during multi-scale fusion. Therefore, in scenarios with complex texture backgrounds and coexistence of multi-scale defects, there may still be problems such as insufficient feature representation, fixed fusion weights, or insufficient adaptability.

[0006] Chinese invention patent application CN113628178B, entitled "A Method for Detecting Surface Defects in Steel Products with a Balance of Speed ​​and Accuracy," discloses a target detection model based on CenterNet. This model designs a target detection model for surface defects in hot-rolled strip steel, incorporating skip-layer connections and pyramid feature fusion modules, and outputs detection results after training. This scheme primarily focuses on achieving a balance between detection speed and accuracy, utilizing a pyramid structure to fuse features at different scales. However, this scheme does not disclose cross-axis attention enhancement processing for multi-scale feature maps, nor does it address a normalized fusion mechanism that adaptively learns the contribution of each input feature in the bidirectional feature fusion path. When steel surface defects are small, elongated, low-contrast, or directionally distributed, conventional skip-layer connections and pyramid fusion methods are insufficient to fully highlight the effective response of the defect area, potentially leading to insufficient accuracy in identifying small target defects and low robustness in complex backgrounds.

[0007] Chinese invention patent CN116342542A, entitled "A Method for Detecting Surface Defects in Steel Products Based on Lightweight Neural Networks," discloses a target detection model for surface defects in steel products based on the YOLOv4-tiny model, incorporating an improved pyramid feature fusion module to enhance real-time on-site detection capabilities. This approach emphasizes lightweight networks and real-time on-site detection, making it suitable for resource-constrained scenarios. However, while lightweight backbone networks reduce computational load, they can lead to insufficient representation of fine-grained texture features and long-range contextual information. Without enhancements addressing defect directionality, cross-scale semantic complementarity, and adaptive fusion weights, achieving a balance between accuracy and real-time performance may still be challenging in detecting weak, elongated, and dense defects on steel surfaces.

[0008] Chinese invention patent CN116402763B, entitled "A YOLOv7 Method for Detecting Surface Defects in Hot-Rolled Strip Steel with an Introduced Attention Mechanism," discloses the introduction of a multi-scale fusion lightweight attention module into the feature extraction network of the YOLOv7 network. It also reduces the number of model parameters through grouped convolution to improve the running speed and expressive power of strip steel surface defect detection. This approach demonstrates that attention mechanisms have been used to improve the detection effect of hot-rolled strip steel surface defects. However, its core remains focused on introducing a lightweight attention module into the existing YOLOv7 structure. It does not disclose a feature enhancement method that performs local encoding along the horizontal and vertical directions and further develops global correlation along cross-axis, nor does it establish an end-to-end intelligent analysis link between the detection results and process cause analysis, defect severity assessment, and report generation. Therefore, this approach primarily focuses on the defect detection results themselves at the output level, making it difficult to meet the comprehensive application needs of defect diagnosis, cause tracing, and process improvement suggestions in actual production lines.

[0009] Therefore, while existing deep learning-based steel surface defect detection schemes can replace manual inspection and improve detection efficiency to some extent, they still have the following shortcomings: First, some schemes mainly rely on conventional convolutional networks or ordinary attention modules, which do not fully utilize the features of steel surface defects such as extension along different directions, weak local texture, and strong background perturbation, resulting in insufficient stability in the recognition of slender defects, small-scale defects, and low-contrast defects. Second, although some schemes adopt multi-scale feature fusion structures, the contribution relationship between features of different scales usually lacks effective learnable normalization constraints, making it difficult to dynamically adjust the fusion weights according to the defect scale and image content, which can easily lead to an imbalance in the utilization of shallow detail information or deep semantic information. Third, some lightweight detection models focus on reducing the number of parameters and improving inference speed, but may sacrifice feature representation ability under complex working conditions, making it difficult to achieve a good balance between real-time performance, accuracy, and robustness. Finally, existing detection systems usually only output defect categories, confidence scores, and location boxes, lacking the ability to further statistically analyze the detection results, assess the severity of defects, diagnose process problems, and automatically generate inspection reports, resulting in a certain disconnect between the detection link and quality management and process optimization.

[0010] In summary, overcoming the problems of insufficient expression of directional features of steel surface defects, weak adaptive ability of multi-scale feature fusion, insufficient robustness of lightweight detection models under complex working conditions, and lack of intelligent diagnostic analysis of detection results in existing technologies, and proposing a more accurate, stable, robust, and easy-to-deploy end-to-end detection method, system, and hardware device for steel surface defects based on deep learning, has become a technical challenge that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0011] To address the shortcomings of existing technologies, such as insufficient representation of directional features of steel surface defects, weak adaptive capability of multi-scale feature fusion, insufficient robustness in detecting small-scale and low-contrast defects under complex working conditions, and difficulty in further using detection results for defect diagnosis and report generation, this invention proposes an end-to-end detection method for steel surface defects based on deep learning. This method extracts multi-scale features through a lightweight backbone network, enhances the correlation features of defects in the horizontal and vertical directions using a multi-scale cross-axis attention mechanism, and achieves adaptive fusion of multi-scale features by combining the learnable weight normalization fusion mechanism of a bidirectional feature pyramid network. The detection results are then input into an intelligent analysis module for defect severity assessment, cause analysis, and report generation. This solves the problems of insufficient utilization of defect features, fixed scale fusion weights, difficulty in balancing detection accuracy and real-time performance, and the disconnect between the detection link and quality analysis in existing technologies.

[0012] To achieve the above objectives, this invention provides an end-to-end detection method for steel surface defects based on deep learning, comprising the following steps: S1: Obtain the original surface image of the steel to be inspected; S2: Preprocess the original surface image to obtain the target input image; S3: The target input image is fed into a pre-trained defect detection model for processing to obtain preliminary detection results. These results include at least the defect category, defect confidence score, and defect location information. The defect detection model extracts multi-scale feature maps based on a lightweight backbone network, enhances these features using a multi-scale cross-axis attention mechanism, and then inputs the enhanced feature maps into a bidirectional feature pyramid network for multi-scale feature fusion. Finally, the preliminary detection results are output through the detection head. When fusing multi-scale features in the enhanced feature maps, the bidirectional feature pyramid network employs a learnable weight normalization fusion mechanism. The corresponding fusion calculation formula is as follows:

[0013] in, This represents the output features after fusion; Indicates the first Path input characteristics; Indicates the first Learnable weights for the input features of the path, and ; To prevent the use of a stable constant term with a denominator of zero; Indicates the number of paths or scales of the input features; The index used for weighted summation has a range of values. Same, that is ; Indicates the first Learnable weights corresponding to the input features of the path This represents the summation of the learnable weights corresponding to all input features; S4: Input the preliminary detection results into the intelligent analysis module for comprehensive analysis to obtain the defect diagnosis results, and generate a defect detection report based on the preliminary detection results and the defect diagnosis results.

[0014] As a preferred embodiment of the present invention, step S2 involves preprocessing the original surface image to obtain the target input image, including: Gaussian filtering is applied to the original surface image to remove background noise; Perform contrast enhancement processing on the filtered image; The enhanced image is scaled proportionally to obtain a target input image that meets the input size requirements of the defect detection model.

[0015] As a preferred embodiment of the present invention, step S3 utilizes a multi-scale cross-axis attention mechanism to enhance the features of the multi-scale feature map, specifically including: designating the multi-scale feature map with shape H×W×C as the input feature map. Local enhancement features along the horizontal and vertical directions of the input feature map are extracted through multi-scale local coding branches, and the corresponding calculation formulas are as follows:

[0016]

[0017] in, This represents the features encoded along the horizontal direction. This represents the features encoded along the vertical direction. This indicates the normalization layer processing. Indicates the first in the horizontal direction A one-dimensional convolution operation, Indicates the first in the vertical direction A one-dimensional convolution operation, Indicates the results used for fusing multi-scale coding. Convolution operation.

[0018] As a preferred embodiment of the present invention, step S3, which utilizes a multi-scale cross-axis attention mechanism to enhance the features of the multi-scale feature map, further includes: […]. and The input is fed into the cross-axis global association branch for feature interaction fusion to obtain the enhanced feature map. The corresponding calculation formulas are as follows:

[0019]

[0020]

[0021] in, Indicates interactive output from the top branch. This indicates interactive output from the bottom branch. Indicates will As a query matrix, As a multi-head cross attention along the vertical direction of the key and value matrix Indicates will As a query matrix, Multi-head cross attention along the horizontal direction as a key-value matrix.

[0022] As a preferred embodiment of the present invention, in step S3, the fused features of the 6th layer feature node in the top-down path of the bidirectional feature pyramid network are... and the final output features The expansion calculation formulas are as follows:

[0023]

[0024] in, , These represent the input features from layers 6 and 7, respectively. This represents features derived from the output of a lower layer. This indicates a scale adjustment operation. This indicates convolution processing. and This represents the two-way learnable fusion weights from the top-down node. , and This represents the three learnable fusion weights of the final output node.

[0025] As a preferred embodiment of the present invention, in step S4, the preliminary detection results are input to the intelligent analysis module for comprehensive analysis to obtain defect diagnosis results, including: The preliminary detection results are input into the intelligent analysis module based on a large language model; The intelligent analysis module statistically analyzes and visualizes defect categories, defect confidence levels, and defect location information to assess the severity of surface defects in steel. Based on statistical and evaluation data, the intelligent analysis module diagnoses process problems and outputs defect cause analysis and improvement suggestions as defect diagnosis results.

[0026] A deep learning-based end-to-end steel surface defect detection system, used to execute a deep learning-based end-to-end steel surface defect detection method, includes the following modules: The image acquisition and preprocessing module is used to execute steps S1 and S2 to acquire the original surface image of the steel to be inspected, and to preprocess the original surface image to obtain the target input image. The defect detection inference module executes step S3 by inputting the target input image into a pre-trained defect detection model for processing, obtaining preliminary detection results. These preliminary results include at least the defect category, defect confidence level, and defect location information. Specifically, the defect detection model extracts multi-scale feature maps based on a lightweight backbone network, enhances these feature maps using a multi-scale cross-axis attention mechanism, and inputs the enhanced feature maps into a bidirectional feature pyramid network for multi-scale feature fusion. Finally, the detection head outputs the preliminary detection results. The bidirectional feature pyramid network employs a learnable weight normalization fusion mechanism when fusing multi-scale features in the enhanced feature maps. The AI ​​intelligent analysis and output module is used to execute step S4, inputting the preliminary detection results into the intelligent analysis big model for comprehensive analysis, obtaining the defect diagnosis results, and generating a defect detection report based on the preliminary detection results and the defect diagnosis results.

[0027] A deep learning-based end-to-end steel surface defect detection device, used to execute a deep learning-based end-to-end steel surface defect detection method, comprising: A material conveying unit is used to convey the steel to be tested in step S1; The sensing triggering unit is used in step S1 to detect whether the steel to be detected has reached the detection area and output a trigger signal; The light source unit is used to enhance the light intensity on the surface of the steel to be inspected in step S1. The image acquisition unit is used to capture an image of the steel to be inspected and obtain the original surface image after receiving a trigger signal in step S1. The processing unit is used to receive the original surface image in steps S2, S3 and S4, and sequentially perform image preprocessing, model inference detection and AI intelligent analysis and report generation.

[0028] Compared with the relevant prior art, the beneficial effects of the present invention are: This invention enhances the surface defect features of steel through a multi-scale cross-axis attention mechanism, which can more fully extract the effective features of slender defects, micro-defects and low-contrast defects, and suppress background noise interference such as reflection, oil stains, water stains and oxide scale, thereby improving the accuracy and robustness of steel surface defect detection in complex industrial environments.

[0029] This invention employs a bidirectional feature pyramid network to fuse feature maps at different scales, and adaptively adjusts the contribution of each level of features through a learnable weight normalization fusion mechanism, so that shallow detail features and deep semantic features can be fully combined, which is beneficial to improving the detection effect of small-scale defects such as microcracks, fine scratches, and pits, as well as multi-scale defects.

[0030] This invention employs a lightweight backbone network for feature extraction, which reduces the number of model parameters and computational complexity while ensuring defect recognition capabilities, improves model inference speed, and makes the detection model more suitable for deployment on edge computing devices, embedded terminals, or industrial field computing platforms, meeting the real-time detection needs of high-speed production lines. Attached Figure Description

[0031] Figure 1 The overall flowchart of a deep learning-based end-to-end detection method for steel surface defects is provided in an embodiment of the present invention. Figure 2A schematic diagram of the overall architecture and execution flow of an end-to-end steel surface defect detection system based on deep learning, provided for an embodiment of the present invention; Figure 3 This is a visualization of the NEU-DET dataset provided in this embodiment of the invention; Figure 4 This is a diagram of the improved YOLOv11 model network structure provided in this embodiment of the invention; Figure 5 This is a schematic diagram of a general reverse bottleneck module provided in an embodiment of the present invention; Figure 6 This is a diagram of the feature network structure provided in the embodiments of the present invention; Figure 7 This is a diagram of the EfficientNet structure provided in the embodiments of the present invention; Figure 8 This is an overall structural diagram of the MCA network provided in this embodiment of the invention; Figure 9 This is a detailed structural diagram of the multi-scale cross-axis attention decoder provided in an embodiment of the present invention; Figure 10 These are test effect diagrams of different defects in various models provided in the embodiments of the present invention; Figure 10 (a) is YOLOv11m; Figure 10 (b) is YOLOv11m+MObileNetV4; Figure 10 (c) is YOLOv11m+MObileNetV4+BiFPN; Figure 10 (d) is YOLOv11m+ MObileNetV4+ BiFPN+MCA; Figure 11 This is a diagram of the detection system device provided in an embodiment of the present invention; Figure 12 This is an interface diagram of the developed steel defect detection system provided in an embodiment of the present invention; Figure 13 This is a diagram illustrating the dialogue between the detection system and AI provided in this embodiment of the invention. Figure 14 This is a diagram of a defect analysis report exported by the system provided in an embodiment of the present invention. Detailed Implementation

[0032] The solutions provided by the present invention will be further described below with reference to the accompanying drawings. However, the present invention can be implemented in many different ways and should not be construed as limited to the embodiments shown; rather, these embodiments provide those skilled in the art with implementation methods that meet applicable legal requirements.

[0033] Example 1: The following is combined with Figure 1 and Figure 2 The present invention will provide a more detailed description of the specific execution process of the end-to-end detection method and system for steel surface defects based on deep learning provided by the present invention.

[0034] like Figure 1 The diagram shows the overall flowchart of the end-to-end detection method for steel surface defects based on deep learning according to the present invention. Figure 2 The diagram shows the overall architecture and execution flowchart of a deep learning-based end-to-end steel surface defect detection system according to the present invention. In this embodiment, the end-to-end detection system is deeply integrated into a real industrial hot-rolled strip steel production line, aiming to achieve fully automated closed-loop operation from the arrival of steel materials to the final output of a quality analysis report. Specifically, the detailed execution steps of this embodiment are as follows: Step S1: Obtain the original surface image of the steel to be inspected.

[0035] After system startup, the PLC control unit completes the initialization of all devices and loads the improved YOLOv11 model and DeepSeek AI analysis model deployed on edge computing nodes (such as industrial PCs configured with NVIDIA GeForce RTX 4090 D). The steel sample to be inspected is placed on an industrial conveyor belt, which is set to transport forward at a constant speed of 0.3 m / s. When the steel sample enters the machine vision inspection station, the diffuse infrared sensor (detection distance set to 0-20cm) installed on the side of the conveyor belt captures the infrared signal change at the edge of the steel in real time and immediately sends a rising edge trigger signal (i.e., a hardware interrupt for position counting) to the PLC controller. The PLC responds extremely quickly within 5ms, synchronously triggering two sets of symmetrically arranged LED strip lights (the light source controller is preset to high-brightness strobe mode) and a vertically orthogonally mounted MV-CS016-10UM industrial camera. The industrial camera captures images of a steel surface in high-speed motion under uniform, low-reflectivity illumination from a bar light source at a frame rate of 249.1 fps and an extremely short global shutter exposure time of 1 / 1000 second. This results in a high-resolution, single-channel grayscale raw surface image without motion blur, which is then streamed in real-time to the memory buffer of the edge computing node via the GigE vision protocol.

[0036] Step S2: Preprocess the original surface image to obtain the target input image.

[0037] Upon receiving the original surface image, the image preprocessing module at the edge computing node first performs spatial domain smoothing on the image using a CUDA-accelerated 3×3 Gaussian filter operator on the GPU, effectively removing isolated salt-and-pepper noise points due to the unavoidable dust adhesion and thermal noise from the camera sensor in industrial environments. Then, for low-contrast defects commonly found on hot-rolled strip steel surfaces (such as shallow pitted surfaces), the system employs a histogram equalization-based contrast enhancement algorithm to stretch the image's grayscale dynamic range and highlight the edge gradients between defects and the background. Finally, to meet the strict alignment requirements of the tensor dimensions in deep learning models, a bilinear interpolation algorithm is used to proportionally scale the enhanced image. If the original image's aspect ratio is not 1:1, a grayscale padding operation with a pixel value of 128 is performed on both sides of the shorter side, resulting in a final output image with a strictly fixed resolution of 640×640×1.

[0038] Step S3: Input the target input image into the pre-trained defect detection model for processing to obtain preliminary detection results.

[0039] The target input image is converted into a floating-point tensor and fed into the improved defect detection model (i.e., YOLOv11 framework + MobileNetV4 backbone + BiFPN neckline + MCA attention module). The detailed process of feature transformation is as follows: (1) Multi-scale feature extraction of backbone network: The input tensor with size 640×640×1 first enters the MobileNetV4ConvSmall lightweight backbone network. After dimensionality reduction by 3×3 convolution (stride of 2) in Conv0 layer, it passes through the Universal Inverted Bottleneck (UIB) modules from Layer 1 to Layer 4 in sequence. The backbone network downsamples and extracts features layer by layer, and finally outputs multiple sets of feature maps with different receptive fields to the next stage, including shallow high-resolution features F1, mid-level features F2, mid-deep features F3, and deep high-level semantic features F4 (size reduced to 20×20×128). Among them, the deep feature F4 is then enlarged globally and encoded with location information by SPPF (spatial pyramid pooling) and C2PSA components.

[0040] (2) MCA Multi-Scale Cross-Axis Feature Enhancement: To address the issue of weak and easily lost features in extremely small defects (such as crazing cracks, which often only occupy a few pixels), the features output by C2PSA are then fed into the MCA module. Assume the feature map input to the MCA module... There is a tiny, low-contrast horizontal scratch defect. First, MCA uses multi-scale local coding branches and 1×7, 1×11, and 1×21 one-dimensional convolutions to scan features along the horizontal (x-axis). Since the scratch is horizontal, the horizontal filter has a strong response, resulting in the calculated... The representative scalar value of the activation vector in the scratch region spikes to 0.85; while the 11×1 one-dimensional convolution along the vertical (y-axis) yields... The activation value is only 0.20. Then, it enters the cross-axis global association branch. Based on vertical features... As a query, it features a strong horizontal response. Multi-head cross-attention calculation is performed using keys and values. This mechanism allows contextual information in the vertical dimension to be guided and weighted by strong features in the horizontal dimension, as shown by the formula. After fusion, the final output activation value of the tiny scratched area is nonlinearly amplified to 0.92, while the background noise of the surrounding defect-free normal texture is suppressed to below 0.05, achieving absolute enhancement of local details.

[0041] (3) BiFPN Adaptive Weighted Feature Fusion: Deep features enhanced by MCA (denoted as P5 node) and mid-to-shallow features (P4 and P3 nodes) are fused bidirectionally into the BiFPN network. The core of BiFPN lies in the use of a learnable weight normalization formula: The specific fusion process is as follows: When the model is processing the "minor lateral scratch" enhanced by MCA, the fusion node needs to combine the deep semantic features transmitted from top to bottom. High-resolution features in the same layer Because scratches are small targets, models trained on large datasets adaptively assign drastically different fusion weights to such targets. For example, the model automatically assigns weights to shallow features. Assign weights to upsampled deep features Assuming The regional scalar value is 1.5. The region has become blurred due to multiple pooling operations, and its scalar value is only 0.3. Substituting this into the formula (setting...) Calculate the fusion output features: .

[0042] (4) Detection Head Output: The final fused multi-scale features are fed into the Anchor-Free detection head to perform classification prediction and bounding box regression. After filtering redundant boxes through non-maximum suppression (NMS), the preliminary detection results are output. The results are presented in a structured data packet, for example: [{"Class": "scratches", "Confidence": 0.87, "BBox":[120, 50, 185, 290]}, {"Class": "inclusion", "Confidence": 0.65, "BBox":[400, 310, 420, 330]}].

[0043] Step S4: Input the preliminary detection results into the intelligent analysis module for comprehensive analysis, obtain the defect diagnosis results, and generate a report.

[0044] At the system software level, the PyQt framework was used to build the human-computer interaction interface (such as...). Figure 2 (As shown in the backend logic). Edge nodes push the aforementioned structured data packets to the UI interface in real time for highlighted bounding box rendering. Simultaneously, the data is synchronously pushed to the embedded DeepSeek AI intelligent analysis model. During the continuous detection period in this embodiment, DeepSeek AI summarizes the statistical data of the past 2440 frames (e.g., a total of 6678 defects, of which 4608 are scratches, accounting for 69%, and the average confidence level is >0.63). Based on this statistical result, DeepSeek AI activates the expert knowledge reasoning graph, diagnoses and outputs the process problem analysis text: "The main defect is linear scratches, which show a high frequency and high confidence concentrated outbreak. The cause is inferred to be: severe mechanical damage or steel adhesion on the surface of the rolls or guide devices; hard protrusions on the conveying track causing continuous scratching. It is recommended to immediately take emergency shutdown measures for the entire line of equipment for inspection and verify the status of the guide plates and pinch rolls." Ultimately, the system not only displays the initial image detection frame to the quality inspector, but also packages and merges the AI's diagnostic conclusions, various defect distribution pie charts, severity assessment bar charts, etc., and automatically generates and exports an unalterable standardized PDF "Steel Defect Detection and Analysis Report", completing the end-to-end detection closed loop.

[0045] Example 2: This example focuses on the specific network architecture details of the pre-trained defect detection model of the present invention, the preparation and processing of the experimental dataset, the specific hyperparameter settings for model training, and the ablation experiments and test results comparison conducted to verify the effectiveness of the present invention.

[0046] The dataset used in this invention is the NEU-DET dataset, which collects six typical surface defects of hot-rolled strip steel. The dataset contains a total of 1800 images of steel surface defects, divided in an 8:2 ratio, with 1419 images in the training set and 381 images in the test set. It includes six categories: crazing, inclusion, patches, pitted surface, rolled-in scale, and scratches. Simultaneously, Mosaic data augmentation techniques (rotation, scaling, flipping, etc.) are used to expand the dataset, addressing the issues of insufficient defect sample quantity and uneven distribution, thereby improving the model's generalization ability.

[0047] like Figure 3 As shown, this is a visualization of the NEU-DET dataset provided in this embodiment of the invention, which allows for a direct view of the images in part of the dataset. The industrial defect dataset has the following characteristics: (1) The images are all grayscale images, so there is no need to convert them to grayscale, which can save training time and allow them to be directly input into the model for processing. (2) Industrial defects exist in various shooting states. The images in this dataset contain scratches in various directions and images taken from various angles within the same category, which can extract the features of the target object as much as possible.

[0048] After preparing the dataset, the images from the training set are input into the improved YOLOv11 model for training. To optimize model performance and achieve the core goals of lightweight, high-precision detection, while also accelerating model inference capabilities and facilitating deployment on industrial edge devices, this invention makes targeted improvements to the native YOLOv11 model. For example... Figure 4 The diagram shown is a network structure diagram of the improved YOLOv11 model provided in this embodiment of the invention. This invention uses YOLOv11 as the basic framework, replacing the original baseline model's CSPDarknet backbone network with a lightweight MobileNetV4 backbone network before connecting it to the SPPF layer. This significantly reduces the number of model parameters and improves inference speed while retaining the C2PSA components in YOLOv11, providing support for edge deployment of the model. Simultaneously, a BiFPN bidirectional feature pyramid replaces the original PANet feature fusion structure, and an MCA multi-scale cross-axis attention module is introduced after feature fusion.

[0049] Lightweight Backbone Network (MobileNetV4) Design: MobileNetV4 is a next-generation lightweight convolutional neural network architecture specifically designed for edge computing and low-latency scenarios. Its core design philosophy is to minimize computation and parameter count while maintaining feature extraction capabilities, thereby achieving efficient inference. This invention uses the MobileNetV4 ConvSmall version as the backbone network, its core being the introduction of a Universal Inverted Bottleneck (UIB) search block. For example... Figure 5 The diagram shows the general inverse bottleneck module principle provided in this embodiment of the invention. UIB is a unified and flexible structure that extends the Inverted Bottleneck (IB) module of MobileNet by introducing optional depthwise separable convolutions (DWs) before the extension layer and between the extension layer and the projection layer. Adding it unifies IB, ConvNext, Feedforward Network (FFN), and the novel ExtraDW variant. In this embodiment, the MobileNetV4ConvSmall backbone network has a total of 5 layers: the first three layers use ConvBN to extract basic feature information; Layers 3 and 4 improve the computational efficiency and multi-scale feature extraction capabilities of the network through depthwise convolutions. The specific network structure is shown in Table 1.

[0050] Table 1: Specific Structure of MobileNetV4ConvSmall Backbone Network

[0051] While ensuring feature extraction capabilities, this structure has only about 2.8 million backbone parameters, which is more than 60% lower than the native YOLOv11m model. The computational load is reduced by about 60%, and the inference speed is increased by more than 40%, making it perfectly suited for the deployment needs of industrial edge devices.

[0052] This embodiment uses a BiFPN bidirectional feature pyramid instead of the native PANet feature fusion structure to enhance the fusion effect of multi-scale features, focusing on improving the feature representation capability of small-sized defects. For example... Figure 6 The diagram shows the feature network structure provided in this embodiment of the invention. To overcome the limitations of the unidirectional information flow in traditional FPN, PANet adds a bottom-up path aggregation network, while BiFPN, based on PANet, removes nodes with only one input edge and adds some skip connections, achieving cross-layer feature fusion, improving information flow, and reducing network complexity. Figure 7The diagram shown is an EfficientNet structure provided in this embodiment of the invention, demonstrating how BiFPN constructs a bidirectional feature transfer structure through top-down upsampling and bottom-up downsampling paths. BiFPN introduces learnable weights to achieve adaptive weighted feature fusion, and its core weighted fusion formula is:

[0053] in, The learnable weights for the features of each branch, Input features at different scales, and then... Applying the ReLU function afterwards ensures , This is a very small value used to avoid numerical instability. This method ensures that the value of each normalized weight falls between 0 and 1, thereby achieving efficient and stable feature fusion. During the network's forward pass, high-level strong semantic features are upsampled and then weighted and fused with low-level high-resolution detail features. The enhanced local features are then downsampled and fed back to the higher levels, forming a cyclically enhanced bidirectional information flow. This is done in the top-down path. Taking node feature fusion as an example, the calculation formula is as follows:

[0054]

[0055] Taking the top branch as an example, given the feature map The encoding process is as follows:

[0056] Output of the bottom branch Similarly:

[0057] Then calculate and The cross-attention between them is calculated as follows:

[0058]

[0059] The final output of the MCA network's multi-scale cross-axis attention can be expressed as:

[0060] To verify the effectiveness of each improved module, ablation experiments were conducted using the native YOLOv11m as the baseline model. Specific comparisons are shown in Table 2. The ablation experiments demonstrate that the improved final model of this invention (YOLOv11m+MobileNetV4+BiFPN+MCA) reduces the number of parameters by 35.5%, computation by 42.5%, and model size by 34.7% compared to the baseline YOLOv11m model. The average accuracy (mAP@0.5) increases from 74.6% to 82.3% (an improvement of 7.7%). Although the frame rate slightly decreases (from 322 FPS to 286 FPS), it maintains high detection accuracy while significantly reducing computation and parameters, achieving an effective balance between lightweight design and accuracy.

[0061] Table 2: Comparison of Ablation Experiment Results of Target Detection Model like Figure 10 The image shows a comparison of the test results of different defects provided in various models in embodiments of the present invention. Wherein: Figure 10 (a) is the YOLOv11m baseline model; Figure 10 (b) shows the result of incorporating MobileNetV4; Figure 10 (c) shows the results after adding BiFPN; Figure 10 (d) shows the detection results of the final model of this invention. A comparison clearly reveals that the baseline model has low confidence in defects and insufficient localization accuracy; the final model, after incorporating the MCA attention module, achieves optimal confidence and bounding box localization accuracy for various defects (such as the highly challenging crazing cracks and pitted surfaces). It significantly improves the recognition ability of low-contrast, subtle defects, effectively suppresses background noise interference, and thoroughly verifies the high effectiveness of the various hardware and software collaborative improvement mechanisms in steel inspection tasks.

[0062] Example 3: To further illustrate the application of the technical solution of the present invention in actual industrial scenarios, this example focuses on the end-to-end closed-loop integrated innovation of the present invention, namely, the specific implementation of the image acquisition hardware device, software functional modules and large model interactive interface.

[0063] The hardware selection for this invention is based on mature industrial-grade products, ensuring cost control, strong environmental adaptability, and stable system operation. For example... Figure 11The diagram shows the detection system device provided in this embodiment of the invention. The hardware detection device is mainly composed of an industrial camera, a strip light source, a conveyor belt, an infrared sensor, and a PC. Specifically, it includes: (1) Image acquisition unit: an MV-CS016-10UM black and white global shutter industrial camera, which is vertically installed above the vision support. The camera supports a high frame rate of 249.1fps and a resolution of 10 million pixels. Its parameters can perfectly meet the high requirements of steel surface defect detection for image quality. The camera is widely used in the field of industrial inspection, has a mature technical system and stable performance, and can adapt to the complex working conditions in the steel production environment, providing reliable guarantee for image acquisition. (2) Light source unit: includes two sets of LED strip light sources and matching light source controllers, which are symmetrically arranged on both sides of the industrial camera. The height and illumination angle are adjustable, aiming to provide low-reflection uniform illumination for the detection area (brightness can be adjusted in real time by the light source controller). The design of the strip light source can effectively improve the illumination conditions of the steel surface and reduce the interference of metal reflection, shadows, etc. on image quality. By arranging the light source reasonably, the weak defect features on the steel surface can be highlighted, providing clear and accurate image data for subsequent defect identification and analysis. (3) Transmission unit: An industrial uniform speed conveyor belt is used, with a running speed that is continuously adjustable in the range of 0.1m / s to 0.5m / s (controlled by a speed controller), and the belt surface is non-slip and wear-resistant. The running speed and stability of the conveyor belt directly affect the efficiency and accuracy of the detection. This conveyor belt can realize the stable transmission of steel samples, so that the steel passes under the camera's field of view at a uniform speed, ensuring that the industrial camera and infrared detection device can perform comprehensive and continuous detection of the steel surface, avoiding missed detection or false detection due to transmission jitter. (4) Sensing trigger unit: A diffuse reflection infrared sensor is used, with the detection distance set between 0 and 20cm. This device uses the infrared signal change characteristics caused by the steel passing through the detection area to capture and analyze the infrared sensing signal, realizing the function of detecting the steel in place and automatically triggering the camera to collect data and the system to count. (5) Processing unit and human-computer interaction unit: The processing unit is equipped with a high-performance PC (equipped with an NVIDIA GeForce RTX-level GPU), and internally deploys an improved YOLOv11 target detection model and a DeepSeek AI analysis module, which can quickly process a large amount of image data collected by industrial cameras and work together to complete real-time inference and deep analysis; at the same time, it works with a PC monitor to form a human-computer interaction unit, which displays detection results, configures parameters and manages data in real time.

[0064] This invention also developed an end-to-end steel defect detection system software that can be used in conjunction with the invention, and independently designed a human-computer interaction interface using PyQt technology. For example... Figure 12The diagram shows the interface of the steel defect detection system developed in this embodiment of the invention. This system integrates real-time detection, AI analysis, data statistics, and report export, and specifically includes the following modules: (1) Image acquisition and preprocessing module: used to receive steel surface images streamed from an industrial camera in the background, and perform preprocessing operations such as image noise reduction, contrast enhancement, and size normalization to provide high-quality input for defect detection. (2) Defect detection module: equipped with the aforementioned improved YOLOv11 lightweight target detection model, used to identify typical defects such as cracks, scratches, inclusions, pitted surfaces, and patches in real time. Figure 12 On the left side, in the "Real-time Detection Screen", the system intuitively outputs the defect boundary location box, defect category name and confidence value. At the same time, the system supports three flexible detection methods: image detection, real-time camera detection and video detection. (3) Data statistics and management module: used to count key indicators such as the total number of defects, the proportion of each type of defect, confidence distribution, and detection frame rate in real time, and display them in the status bar at the bottom of the interface in real time (such as "Detection completed, X defects in total"), and store and trace the detection results. (4) Report generation and export module: based on the detection and AI analysis results, automatically generate a standardized PDF report containing defect details, statistical charts and AI diagnostic conclusions, and support one-click export through the "Export PDF Report" button at the bottom of the interface, which is convenient for workshop quality control and archiving.

[0065] One of the core innovations of this software system lies in the introduction of a large language model to assist decision-making. The system interface features a "Control Panel" on the right, which includes a dedicated "AI Intelligent Analysis" area. This module is deeply integrated with the defect detection module, enabling it to automatically perform defect cause analysis and severity assessment, and provide process diagnosis through natural language interaction. Furthermore, users can directly interact with DeepSeek AI within the interface, achieving intelligent answers and analysis for relevant professional questions. Figure 13The diagram illustrates the dialogue between the detection system and AI provided in this embodiment of the invention. Specifically, after image detection is completed, if several defects are found, the operator can input custom requirements in the AI ​​question box, such as: "Analyze the causes of the above defects and propose solutions," and then click the "Ask AI" command. Upon receiving the prompt, DeepSeek AI will immediately combine the structured data captured from the current image (e.g., a total of 5 defects were found, all of type "crazing," with confidence levels between 0.31 and 0.77), and output professional analysis and diagnosis in the results display area below. For example, the AI ​​clearly points out that "crazing is the main and clear defect pattern present on the surface of the steel plate," and further provides cause inferences and targeted solution suggestions from dimensions such as billet quality, furnace temperature control, and abnormal rolling processes. In this way, the present invention upgrades isolated "image defect recognition" to "process closed-loop guidance," greatly improving the intelligence level and field application value of the end-to-end detection system.

[0066] like Figure 14 The image shows a defect analysis report exported by the system provided in this embodiment of the invention. This report is automatically generated by the system after image detection, defect statistics, and AI intelligent analysis, and is used for centralized display and archiving of the detection results. The upper part of the report includes a "Detection Result Summary" area, which displays the quantity and proportion of high-quality defects, defects requiring review, and serious defects in a card-like format, enabling operators to quickly grasp the current surface quality status of the steel. The middle part of the report includes an "AI Analysis Results" area, where the system automatically generates analysis of major defect types, severity assessments, cause inferences, and improvement suggestions based on structured data such as defect categories, defect confidence levels, defect quantities, and defect distribution output by the detection model. The AI ​​intelligent analysis module can comprehensively assess different defect types such as scratches, pitting, inclusions, and cracks from aspects such as rolling process, conveying and handling process, heating and cooling conditions, raw material quality, and equipment status, and output targeted process optimization suggestions. The bottom of the report further records information such as the report generation time, detection system name, and system version, facilitating subsequent quality traceability, production review, and process improvement. Through the report export function, this invention can output standardized documents containing real-time detection results, defect statistics, and intelligent diagnostic conclusions, realizing closed-loop management from defect identification and analysis to quality report generation, and improving the readability, traceability, and engineering application value of steel surface defect detection results.

[0067] The above embodiments merely illustrate implementation methods of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A deep learning-based end-to-end method for detecting surface defects of steel, characterized in that: Includes the following steps: S1: Obtain the original surface image of the steel to be inspected; S2: Preprocess the original surface image to obtain the target input image; S3: The target input image is fed into a pre-trained defect detection model for processing to obtain preliminary detection results. These results include at least the defect category, defect confidence score, and defect location information. The defect detection model extracts multi-scale feature maps based on a lightweight backbone network, enhances these maps using a multi-scale cross-axis attention mechanism, and then inputs the enhanced feature maps into a bidirectional feature pyramid network for multi-scale feature fusion. Finally, the preliminary detection results are output through the detection head. When fusing multi-scale features in the enhanced feature maps, the bidirectional feature pyramid network employs a learnable weight normalization fusion mechanism. The corresponding fusion calculation formula is as follows: ； in, This represents the output features after fusion; Indicates the first Path input characteristics; Indicates the first Learnable weights for the input features of the path, and ; To prevent the use of a stable constant term with a denominator of zero; Indicates the number of paths or scales of the input features; The index used for weighted summation has a range of values. Same, that is ; Indicates the first Learnable weights corresponding to the input features of the path This represents the summation of the learnable weights corresponding to all input features; S4: Input the preliminary detection results into the intelligent analysis module for comprehensive analysis to obtain the defect diagnosis results, and generate a defect detection report based on the preliminary detection results and the defect diagnosis results.

2. The end-to-end detection method for steel surface defects based on deep learning according to claim 1, characterized in that: Step S2 involves preprocessing the original surface image to obtain the target input image, including: Gaussian filtering is applied to the original surface image to remove background noise; Perform contrast enhancement processing on the filtered image; The enhanced image is scaled proportionally to obtain a target input image that meets the input size requirements of the defect detection model.

3. The end-to-end detection method for steel surface defects based on deep learning according to claim 1, characterized in that: Step S3 utilizes a multi-scale cross-axis attention mechanism to enhance the features of the multi-scale feature map. Specifically, this includes: denoting the multi-scale feature map with shape H×W×C as the input feature map. Local enhancement features along the horizontal and vertical directions of the input feature map are extracted through multi-scale local coding branches, and the corresponding calculation formulas are as follows: ； ； in, This represents the features encoded along the horizontal direction. This represents the features encoded along the vertical direction. This indicates the normalization layer processing. Indicates the first in the horizontal direction A one-dimensional convolution operation, Indicates the first [number] along the vertical direction A one-dimensional convolution operation, Indicates the results used for fusing multi-scale coding. Convolution operation.

4. The end-to-end detection method for steel surface defects based on deep learning according to claim 3, characterized in that: Step S3, which utilizes a multi-scale cross-axis attention mechanism to enhance the features of the multi-scale feature map, also includes: Will and The input is fed into the cross-axis global association branch for feature interaction fusion to obtain the enhanced feature map. The corresponding calculation formulas are as follows: ； ； ； in, Indicates interactive output from the top branch. This indicates interactive output from the bottom branch. Indicates will As a query matrix, As a multi-head cross attention along the vertical direction of the key and value matrix Indicates will As a query matrix, Multi-head cross attention along the horizontal direction as a key-value matrix.

5. The end-to-end detection method for steel surface defects based on deep learning according to claim 1, characterized in that: In step S3, for the 6th layer feature node in the top-down path of the bidirectional feature pyramid network, its fused features are... and the final output features The expansion calculation formulas are as follows: ； ； in, , These represent the input features from layers 6 and 7, respectively. This represents features derived from the output of a lower layer. This indicates a scale adjustment operation. This indicates convolution processing. and This represents the two-way learnable fusion weights from the top-down node. , and This represents the three learnable fusion weights of the final output node.

6. The end-to-end detection method for steel surface defects based on deep learning according to claim 1, characterized in that: In step S4, the preliminary detection results are input into the intelligent analysis module for comprehensive analysis to obtain the defect diagnosis results, including: The preliminary detection results are input into the intelligent analysis module based on a large language model; The intelligent analysis module statistically analyzes and visualizes defect categories, defect confidence levels, and defect location information to assess the severity of surface defects in steel. Based on statistical and evaluation data, the intelligent analysis module diagnoses process problems and outputs defect cause analysis and improvement suggestions as defect diagnosis results.

7. A deep learning-based end-to-end steel surface defect detection system, used to perform the method as described in any one of claims 1 to 6, characterized in that: Includes the following modules: The image acquisition and preprocessing module is used to execute steps S1 and S2 to acquire the original surface image of the steel to be inspected, and to preprocess the original surface image to obtain the target input image. The defect detection inference module executes step S3 by inputting the target input image into a pre-trained defect detection model for processing, obtaining preliminary detection results. These preliminary results include at least the defect category, defect confidence level, and defect location information. Specifically, the defect detection model extracts multi-scale feature maps based on a lightweight backbone network, enhances these feature maps using a multi-scale cross-axis attention mechanism, and inputs the enhanced feature maps into a bidirectional feature pyramid network for multi-scale feature fusion. Finally, the detection head outputs the preliminary detection results. The bidirectional feature pyramid network employs a learnable weight normalization fusion mechanism when fusing multi-scale features in the enhanced feature maps. The AI ​​intelligent analysis and output module is used to execute step S4, inputting the preliminary detection results into the intelligent analysis big model for comprehensive analysis, obtaining the defect diagnosis results, and generating a defect detection report based on the preliminary detection results and the defect diagnosis results.

8. A deep learning-based end-to-end steel surface defect detection device, used to perform the method as described in any one of claims 1 to 6, characterized in that: include: A material conveying unit is used to convey the steel to be tested in step S1; The sensing triggering unit is used in step S1 to detect whether the steel to be detected has reached the detection area and output a trigger signal; The light source unit is used to enhance the light intensity on the surface of the steel to be inspected in step S1. The image acquisition unit is used in step S1 to capture the steel to be inspected by the industrial camera after receiving the trigger signal, and to obtain the original surface image. The processing unit is used to receive the original surface image in steps S2, S3 and S4, and sequentially perform image preprocessing, model inference detection and AI intelligent analysis and report generation.

Images