A steel surface defect detection method based on RT-DETR

Abstract

The application discloses a steel surface defect detection method based on RT-DETR, which comprises the following steps: taking an RT-DETR model as a baseline model to construct a defect detection model; inputting a steel surface image into a backbone network to extract features P3, P4 and P5, and enhancing P5 to obtain a feature Y5; inputting P3, P4 and Y5 into a pyramid context extraction module to obtain features P3', P4' and Y5'; inputting P3, P4 and Y5 into a rectangular self-calibration module to obtain features P3'', P4'' and Y5''; performing first fusion on Y5'' and Y5' to obtain M5; performing second fusion on P4'' and M5, and then performing first fusion on P4' and M5 to obtain M4; performing second fusion on P3'' and M4, and then performing first fusion on P3' and M4 to obtain M3; performing down-sampling on M3 and third fusion on M3 and M4 to obtain N4, performing down-sampling on N4 and third fusion on N4 and N5 to obtain N5; and inputting M3, N4 and N5 into a detection head to perform defect detection. The application can solve the problems that the precision and speed of the steel surface defect detection in the prior art are difficult to be considered together, and the detection capability for weak defects and multi-scale defects is insufficient.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method for detecting surface defects in steel based on RT-DETR. Background Technology

[0002] Surface defects in steel not only affect product appearance but also represent potential weaknesses in material performance, directly impacting the reliability and lifespan of major engineering projects and everyday consumer goods. Traditional inspection methods relying on manual visual inspection or conventional machine vision suffer from low efficiency, high false negative rates, and susceptibility to subjective factors, making them unsuitable for high-speed production lines. Photoelectric detection, based on optical and electromagnetic induction principles, offers improvements in detection speed and measurement accuracy compared to manual methods, but it relies on precision sensors, requires high environmental stability, and each detection method is typically designed for specific defects, resulting in considerable maintenance costs. Traditional machine learning methods follow a feature engineering-driven path, heavily relying on manually designed features, and lack robustness and generalization ability when facing unknown defects or complex conditions. Deep learning-based object detection is divided into single-stage and two-stage methods: single-stage methods (such as YOLO, SSD, and RetinaNet) directly output results in a single forward computation, offering fast inference speed but typically lower accuracy than two-stage methods; two-stage methods (such as Faster R-CNN and SPPNet) first generate candidate boxes and then refine them, offering high accuracy but slower speed.

[0003] The "Online Detection Method, Apparatus, System, and Storage Medium for Steel Surface Defects" disclosed in Chinese patent literature, publication number CN121453831A, published on February 3, 2026, includes: acquiring a preset reference spot map; wherein the reference spot map is obtained by establishing a coordinate system from a first aluminum nitride spot image on the surface of a defect-free steel sample under a scanning electron microscope; after the target steel leaves the annealing furnace, using the residual heat of the target steel, raised aluminum nitride spots are generated on the surface of the target steel through a pulse nitriding process; acquiring a second aluminum nitride spot image on the surface of the target steel, transforming the second aluminum nitride spot image into a coordinate system, and generating an online spot map; using the reference spot map as prior knowledge for a pre-trained defect recognition model, and inputting the online spot map into the defect recognition model to identify defects in the target steel. However, this technology is still a single-stage deep learning object detection method, which suffers from fast inference speed but lack of accuracy. Summary of the Invention

[0004] This invention aims to overcome the problems in existing technologies where it is difficult to balance accuracy and speed in steel surface defect detection, and where the detection capabilities for weak defects and multi-scale defects are insufficient. It provides a steel surface defect detection method based on RT-DETR.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for detecting surface defects in steel based on RT-DETR, comprising: A defect detection model is constructed using the RT-DETR model as the baseline model. The steel surface image is input into the backbone network, features P3, P4 and P5 are extracted, and feature Y5 is obtained by enhancing P5. P3, P4, and Y5 are input into the pyramid context extraction module to obtain features P3', P4', and Y5'. P3, P4, and Y5 are input into the rectangular self-calibration module to obtain features P3”, P4”, and Y5”. Y5” and Y5' are first fused to obtain M5; P4” and M5 are second fused, and then P4' is first fused to obtain M4; P3” and M4 are second fused, and then P3' is first fused to obtain M3; M3 is downsampled and then fused with M4 to obtain N4. N4 is downsampled and then fused with N5 to obtain N5. M3, N4, and N5 are fed into the detection head for defect detection.

[0006] Preferably, the process of enhancing P5 to obtain feature Y5 employs the intra-scale feature interaction module AIFI-WPA based on dual-domain attention, including: The frequency domain branch extracts high and low frequency features through wavelet transform decomposition, and after weighted fusion by MSCM, multi-head self-attention is performed to obtain the first output; The prototype branch generates prototype features through depthwise separable convolution and dynamic clustering, and then obtains the second output after weighting by MSCM; Add the outputs of the two branches to get Y5.

[0007] Preferably, the frequency domain branching process includes: The input features are decomposed using Haar wavelet decomposition to obtain low-frequency and high-frequency sub-bands, and then flattened to obtain high- and low-frequency features. Weights are generated through a multi-scale context module, and high- and low-frequency features are then fused using weighted methods. The fusion result is used as the key and value, and the original query Q is used as the query. Multi-head self-attention is performed, and the first output is obtained through residual connection and layer normalization.

[0008] Preferably, the processing of the prototype branch includes: The input features are subjected to depthwise separable convolution and flattened into a sequence; The sequence generates a multi-prototype weight matrix through linear mapping, and calculates the prototype and interpolates the source features of the sub-band sum after wavelet decomposition. The gating weights are obtained through the multi-scale context module, and the residual weighting is performed to obtain the second output.

[0009] Preferably, the processing procedure of the pyramid context extraction module is as follows: The input features P3, P4, and Y5 are adaptively pooled to a uniform minimum size to obtain pooled features. All pooling features are concatenated along the channel dimension to obtain the aggregated feature Z; Z is then processed through several rectangular self-calibrating attention modules. The processed features are split according to the original number of input channels to obtain enhanced features P3', P4', and Y5'.

[0010] Preferably, the processing procedure of the rectangular self-calibration module is as follows: The input feature rectangle self-calibration attention mechanism extracts axial global context information and performs normalization. The normalized features are input into a convolutional multilayer perceptron with GELU activation and then multiplied by the layer scaling parameter. The scaled features are added to the original input features to obtain the output features.

[0011] Preferably, the first fusion is a cross-scale gated fusion, in which spatial attention weights are generated by 1×1 convolution and h-sigmoid activation of high-level features, and low-level features are modulated element-wise after upsampling. The second fusion is a bilinear additive fusion, which involves adjusting the channels of the upsampled high-level features and adding them element-by-element to the low-level features.

[0012] Preferably, the downsampling includes: The offset prediction convolutional layer generates 2N offsets for each spatial location, where N is the predefined number of sampling points; The sampling grid coordinates are added with an offset to obtain the deformed sampling position, and bilinear interpolation is performed at the integer coordinate points around it to collect N feature vectors; rearrange them along the height direction so that the N sampling points at the same spatial position are stacked continuously, and then input into a standard convolutional layer to obtain the downsampled feature map.

[0013] Preferably, the third fusion is a fast normalization fusion, which assigns learnable weights to each input feature, obtains the corresponding fusion coefficients after ReLU and normalization, and then performs a weighted summation.

[0014] Preferably, the method further includes model training and evaluation steps: Obtain a sample set of surface defects in steel and divide it into a training set, a validation set, and a test set; The defect detection model is trained iteratively using a pre-defined training set. Training is stopped and the model parameters are saved when the number of training rounds reaches the preset number of rounds. Based on the validation set, the model performance index is evaluated, and the best performing model on the validation set is determined as the optimal defect detection model.

[0015] This invention has the following beneficial effects: It employs a dual-domain attention-based intra-scale feature interaction module to separate high and low frequencies to enhance weak defect edges in the frequency domain and suppress background noise in the prototype domain, thereby improving sensitivity to minute defects while retaining global modeling capabilities; the fusion network achieves refined fusion of multi-scale features through the synergy of a pyramid context extraction module, a rectangular self-calibration module, bilinear additive fusion, cross-scale gated fusion, and fast normalization fusion, avoiding channel expansion caused by simple splicing and improving the robustness of multi-size defect detection in complex backgrounds; linear deformable convolution LDConv replaces fixed convolution downsampling, adaptively adjusting the receptive field along the defect curvature, reducing redundant sampling, and preserving key spatial structures. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a steel surface defect detection method based on RT-DETR in this invention.

[0017] Figure 2 This is a schematic diagram of cross-scale gating fusion and bilinear additive fusion in this invention.

[0018] Figure 3 This is a schematic diagram of the rectangular self-calibration module in this invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be noted that all data in the present invention were collected under the premise of legality and compliance.

[0020] like Figure 1 As shown, a method for detecting surface defects in steel based on RT-DETR includes: A defect detection model is constructed using the RT-DETR model as the baseline model. The steel surface image is input into the backbone network, and multi-scale feature maps P3, P4 and P5 of the steel surface image are generated. P5 is then enhanced to obtain high-level semantic features Y5. Input features P3, P4, and Y5 into the pyramid context extraction module to obtain enhanced features P3', P4', and Y5' with global context information; Features P3, P4, and Y5 are input into the rectangular self-calibration module for processing to obtain features P3”, P4”, and Y5”. Dynamic feature fusion is performed in a top-down order. First, feature Y5” and feature Y5’ are fused together to obtain high-level fused feature M5. Then, feature P4” and feature M5 are fused together, and then feature P4’ is fused together to obtain feature M4. Next, feature P3” and feature M4 are fused together, and then feature P3’ is fused together to obtain feature M3. After downsampling feature M3 and performing a third fusion with feature M4, enhanced mid-layer feature N4 is obtained. After further downsampling feature N4, it is fused with feature N5 to obtain enhanced high-layer feature N5. Send M3, N4, and N5 into the detection head for defect detection, set the number of queries (e.g., 300), the number of decoder layers (e.g., 4), and predict the defect category and bounding box.

[0021] Existing object detection methods mainly fall into two categories: single-stage and two-stage methods. The difference lies in whether an explicit candidate region generation process exists. Mainstream single-stage methods include YOLO, SSD, and RetinaNet. Single-stage methods directly output detection results through a single forward computation, offering high inference speed, but their detection accuracy is generally inferior to two-stage methods. Common two-stage methods include R-CNN, Fast R-CNN, Faster R-CNN, and SPPNet. Two-stage methods first generate candidate boxes and then perform fine-grained classification and position correction. While achieving high detection accuracy, the multi-stage processing results in slower speed. This invention proposes a technical solution to address the difficulty in balancing accuracy and speed in steel surface defect detection, as well as the insufficient detection capability for weak defects and multi-scale defects.

[0022] Specifically, the steel surface defect dataset was first acquired and processed. The Neudet dataset annotations were converted from XML to TXT format. The processed steel surface defect dataset was then randomly divided into training, validation, and test sets in an 8:1:1 ratio. The RT-DETR-r18 model was selected as the baseline model, and a high-precision, low-computational-complexity steel surface defect detection model, WCL-DETR, was constructed. This model includes a backbone network, an AIFI-WPA module (an intra-scale feature interaction module based on dual-domain attention), a multi-scale feature fusion network CGRNFPN, and a detection head.

[0023] Secondly, the intra-scale feature interaction module based on dual-domain attention replaces the multi-head attention mechanism in the AIFI module with a WPA dual-domain attention mechanism based on wavelet enhancement and prototype semantics. WPA employs a dual-branch structure: the frequency domain branch extracts high- and low-frequency features through wavelet decomposition, enhancing sensitivity to weak defect edges and texture details; while the prototype branch dynamically clusters image features to generate prototypes, suppressing background noise and making the query more focused on defect areas. Finally, the outputs of the two branches are fused, utilizing information from the frequency domain (wavelet decomposition) and spatial domain (prototype clustering) to improve the ability to distinguish between weak defects and background noise.

[0024] Then, drawing inspiration from the CGRSeg network and employing Fast Normalized Fusion (FNF), a multi-scale feature fusion network, CGRNFPN, is constructed. CGRNFPN uses a Pyramid Context Extraction (PCE) module to extract and fuse contextual information at different scales from the multi-layer features of the backbone network. After recursive refinement, it outputs more discriminative multi-scale features, thereby improving the robustness and accuracy of detecting multi-size defects in complex backgrounds. Simultaneously, bilinear additive fusion and cross-scale gating fusion are implemented for feature fusion. Bilinear additive fusion is a simple feature overlap addition fusion, upsampling high-level features, adjusting the number of channels, and then adding them element-wise with low-level features, allowing deep semantic information to complement shallow details, thus enhancing feature representation. Cross-scale gating fusion utilizes high-level features to generate spatial attention weights, modulating low-level features element-wise, allowing deep semantic information to guide the activation of shallow details, thereby enhancing the model's ability to discriminate multi-size defects in complex backgrounds. In the bottom-up path aggregation process, fast normalization fusion is used to achieve multi-scale feature reconstruction. Fast normalization fusion introduces learnable normalization weights to perform weighted summation and scale reshaping on feature maps at different levels, thereby achieving adaptive multi-scale feature fusion. This avoids channel expansion and information redundancy caused by simple stitching, while maintaining the efficiency of feature alignment.

[0025] Next, the original downsampling is replaced by linear deformable convolution LDConv. The linear deformable convolution LDConv can adaptively adjust the receptive field sampling points according to the actual geometry of the defect, retain key spatial structural information, and avoid destroying important details.

[0026] Finally, WCL-DETR is iteratively trained using the predefined training set. During this process, the model's performance metrics are evaluated using the validation set. Training is stopped and the model parameters are saved when the predefined number of training epochs is reached. The model with the best performance on the validation set is determined as the optimal WCL-DETR model. The predefined test set is then input into the trained WCL-DETR for defect detection, and the model's recognition performance is evaluated using the test set.

[0027] As a specific implementation, the process of obtaining feature Y5 by enhancing P5 adopts the AIFI-WPA intra-scale feature interaction module based on dual-domain attention, including: frequency domain branch, which extracts high and low frequency features through wavelet transform decomposition, performs multi-head self-attention after weighted fusion by MSCM, and obtains the first output; prototype branch, which generates prototype features through depthwise separable convolution and dynamic clustering, and obtains the second output after weighted by MSCM; the outputs of the two branches are added together to obtain Y5.

[0028] The original AIFI's multi-head self-attention mechanism only models pixel relationships in the spatial domain, making it insensitive to subtle defects with weak textures and low contrast (such as microcracks and point inclusions). It also lacks explicit separation and differential enhancement of high-frequency details (such as steel cracks and scratch edges) from low-frequency background (such as large-scale flat areas). This invention replaces the original AIFI's multi-head attention mechanism with a WPA dual-domain attention mechanism based on wavelet enhancement and prototype semantics. The frequency domain branch extracts high and low-frequency features through wavelet decomposition, enhancing sensitivity to weak defect edges and texture details, while the prototype branch dynamically clusters image features to generate prototypes, suppressing background noise and making the query more focused on defect areas. The WPA module retains the global modeling capabilities of MHSA (Multi-Head Attention Mechanism), achieves computational compression and frequency decoupling through wavelet pooling, introduces depth convolution to extract fine structures, and then aggregates global semantics through prototype aggregation, forming a complementary advantage between the two branches.

[0029] Specifically, the frequency domain branching process includes wavelet pooling decomposition, subband flattening, weight matrix generation, multi-head self-attention execution, and first branch output.

[0030] In wavelet pooling decomposition, the input feature F is decomposed using Haar wavelet to obtain four half-resolution feature subbands, including the low-frequency subband F. LL and high-frequency subband F LH F HL F HH Each sub-band has a size of R^(B×C×H / 2×W / 2), F LL For low-frequency information, F LH F HL F HH These represent high-frequency details in the horizontal, vertical, and diagonal directions, respectively. B is the batch size, C is the number of channels, and H and W are the spatial height and width. Let N = H × W.

[0031] During the subband flattening process, high and low frequency features are obtained through flattening, and the low frequency feature F L Flatten(F) LL )∈R^(B×N / 4×C); The high-frequency feature is the flattened result of the sum of the three high-frequency subbands, F HFlatten(F) LH +F HL +F HH )∈R^(B×N / 4×C).

[0032] During the weight matrix generation process, the weights w1 are generated through the Multi-Scale Context Module (MSCM), and w1 can be represented as σ(MSCM(F)). L +F H )), where σ is the Sigmoid function. Then, the high and low frequency features are weighted and fused to obtain the high and low frequency fused feature Ffre as w1⨀F H +F L , where ⨀ represents element-wise multiplication.

[0033] During the execution of multi-head self-attention, the fusion result Ffre is used as the key and value, and the original query Q is used as the query. Multi-head self-attention A=MultiheadAttention(Q, Ffre, Ffre) is executed. After residual connection and layer normalization LayerNorm(A+Q), the first output X1 is obtained.

[0034] The prototype branch processing includes sequence flattening, weight generation, prototype calculation, and second branch output.

[0035] During the sequence flattening process, a depthwise separable 3×3 convolution (preserving the number of channels C) is applied to the input feature F to enhance the local structure, and then the sequence is flattened into Fconv=Flatten(DSConv3×3(F)).

[0036] During the weight generation process, the flattened sequence Fconv is used to generate a multi-prototype weight matrix M through linear mapping, where M is softmax(Linear(Fconv))∈R^(B×P×C) and P is the number of prototypes.

[0037] In the prototype calculation process, the source feature S is defined as the flattened sequence of the sum of the four sub-bands obtained by wavelet pooling in the frequency domain branch, and S is Flatten(F LL +F LH +F HL +F HH )∈R^(B×N / 4×C); Calculate the prototype Praw using the prototype weight matrix M, where Praw is M T ∙S∈R^(B×P×C); If P and N are not equal, the prototype sequence needs to be interpolated to length N, and Paligned is Interpolate(Praw, Size=N)∈R^(B×N×C).

[0038] During the second branch output process, the gate weight w2 is obtained through the multi-scale context module MSCM, where w2 is σ(MSCM(Paligned+Q)). Then, residual weighting is performed to obtain the second output X2, where X2 is LayerNorm(Q⨀w2+Q), where ⨀ represents element-wise multiplication.

[0039] After obtaining the first output X1 and the second output X2, the sum of the two branch outputs is calculated and restored to the feature map shape Y=Reshape(X1+X2)∈R^(B×C×H×W), thus obtaining the output result of the WPA dual-domain attention mechanism.

[0040] As a specific implementation, the multi-scale feature fusion network CGRNFPN includes a pyramid context extraction module (PCE), a rectangular self-calibration module (RCM), a bilinear additive fusion (BAF), a cross-scale gated fusion (CSGF), and a fast normalized fusion (FNF).

[0041] The multi-scale feature fusion network CGRNFPN borrows from the CGRSeg network and employs Fast Normalization Fusion (FNF). The PCE (Pyramid Context Extraction) module extracts and fuses contextual information at different scales from the multi-layer features of the backbone network. After recursive refinement, it outputs more discriminative multi-scale features, thereby improving the robustness and accuracy of detecting multi-size defects in complex backgrounds. The BAF and CSGF modules are designed for feature fusion. The BAF module is a simple feature overlap addition fusion. After upsampling the high-resolution features and adjusting the number of channels, it is added to the low-resolution features element-wise, so that the deep semantic information and shallow details complement each other, thereby enhancing the feature representation. The CSGF module uses high-resolution features to generate spatial attention weights and modulates the low-resolution features pixel by pixel, so that the deep semantic information guides the activation of shallow details, thereby enhancing the model's ability to discriminate multi-size defects in complex backgrounds. In the bottom-up path aggregation process, fast normalization fusion is used to achieve multi-scale feature reconstruction. By introducing learnable normalization weights, FNF performs weighted summation and scale reshaping on feature maps at different levels to achieve adaptive multi-scale feature fusion, avoiding channel expansion and information redundancy caused by simple stitching, while maintaining the efficiency of feature alignment.

[0042] Channel projection and AIFI (Attention-Based Feature Interaction) processing are performed on the highest-level feature P5 to obtain the initial high-level feature Y5. Then, P3, P4, and Y5 are input into the Pyramid Context Extraction (PCE) module. The processing procedure of the PCE module is as follows: The input features P3, P4, and Y5 are adaptively pooled to a uniform minimum size to obtain pooled features. All pooling features are concatenated along the channel dimension to obtain the aggregated feature Z; Z is then processed through several rectangular self-calibrating attention modules. The processed features are split according to the original number of input channels to obtain enhanced features P3', P4', and Y5'.

[0043] Specifically, let the input feature list X = [X1, X2, ..., Xk], the number of channels of the i-th feature be Ci, and k = 3, corresponding to features {P3, P4, Y5}. The module first adaptively pools each feature to a uniform minimum size using pyramid pooling: H' = ⌊(Hmin-1) / s⌋+1, W' = ⌊(Wmin-1) / s⌋+1, Xk' = AdaptiveAvgPool2d(Xk, (H', W')). Adaptive pooling can preserve effective information in feature maps at different scales and is insensitive to changes in input size. Here, s is the stride (s defaults to 2), and Hmin and Wmin are the spatial dimensions of the last feature.

[0044] All pooled features are concatenated along the channel dimension to form an aggregated feature Z, where Z is Concat(X1', X2', ..., Xk')∈R^(B×Csum×H'×W'), and the total number of channels Csum=∑Ci (i from 1 to k). By concatenating multi-scale features, information is summarized in the same spatial coordinate system, and the local details of low-level features and the semantic abstraction of high-level features are integrated into a tensor.

[0045] The aggregated feature Z is then processed through n consecutive rectangular self-calibrating attention modules (RCA) (n=3) to obtain Z' as the RCA. n ∘RCA n-1 ∘...∘RCA1, where each RCA contains a combination of strip convolution and square convolution, denoted as RCA(X) = Conv band (X) + Conv square (X). After channel relationship enhancement, feature Z' is still a single tensor. It needs to be split into k parts along the channel dimension to restore a feature map with the same number of channels as the input feature layers, so that it can be fused with features of the corresponding scales later. The splitting operation is represented as Yk'=Split(Z', [C1, C2, ..., Ck], dim=1), k=1, 2, ... . Split extracts channel segments of a specified length along the channel dimension in sequence. The output feature list Y'=[Y1', Y2', ..., Yk'], each with a spatial dimension of (H', W'), but the number of channels remains the original Ck.

[0046] The forward propagation of the rectangular self-calibration module (RCM) is defined as Y = X + γ·MLP(Norm(RCA(X))), where Norm is batch normalization, MLP is a convolutional multilayer perceptron with GELU activation, and γ is the learnable layer scaling parameter. RCM is used to enhance P3, P4, and Y5 for subsequent fusion.

[0047] Specifically, such as Figure 3 As shown, the input features are fed into RCA, and the axial global context information is extracted through the rectangular self-calibration attention mechanism to obtain intermediate features enhanced by rectangular region features. The intermediate features are then subjected to batch normalization (Norm) operation to complete the feature standardization process. The normalized features are then input into a convolutional multilayer perceptron (MLP) with GELU activation and multiplied by a learnable layer scaling parameter γ to dynamically adjust the contribution weight of the branch features. The scaled features are then added to the original input features to obtain the output features.

[0048] The first fusion is Cross Scale Gate Fusion (CSGF), which generates spatial attention weights by performing 1×1 convolution and h-sigmoid activation on high-level features, and then upsampling the low-level features element-wise.

[0049] like Figure 2 As shown in (A), for two inputs X1 (size H1×W1) and X2 (size H2×W2, where H1 is greater than H2), the cross-scale gated fusion is represented as Output=Conv 1×1 (X1)⊙interpolate((h sigmoid (Conv 1×1 (X2)), size = H1 × H1), where h sigmoid The h-sigmoid activation is used, and ⊙ represents element-wise multiplication. This enables the generation of spatial attention weights using high-level semantics, which then filters and enhances low-level features.

[0050] The second fusion method is Bilinear Additive Fusion (BAF), which adds the upsampled high-level features to the low-level features element by element after adjusting the channels.

[0051] like Figure 2 As shown in (B), for two inputs X1 (size H1×W1) and X2 (size H2×W2, where H1 is greater than H2), the bilinear addition fusion is represented as Output = X1 + Conv 1×1 (interpolate(X2, size=H1×H1)), where Conv 1×1 This is used to adjust the number of channels in X2 to be the same as that in X1, enabling dynamic residual connections across scales.

[0052] The third fusion is fast normalization fusion, which assigns learnable weights to each input feature, obtains the corresponding fusion coefficients after ReLU and normalization, and then performs weighted summation.

[0053] To endow the bottom-up path with the ability to dynamically balance multi-source features, Fast Normalized Fusion (FNF) is used to replace the original Concat operation. FNF originated from the idea of ​​weighted feature fusion, but avoids the computational overhead and training instability caused by softmax normalization. Its core is a set of learnable scalar weights and its fast normalization mechanism.

[0054] For a given set of m input features {I1, I2, ..., Im} (m=2, representing features downsampled in the previous stage and top-down features fused at the current scale, respectively), FNF first unifies the number of feature channels to the same dimension C using a 1×1 convolution. Then, it assigns a learnable scalar weight ωi∈R to each input. The weights are then activated by ReLU and normalized to obtain the fusion coefficient ωi. norm ,ωi norm Let ReLU(ωi) / (∑ReLU(ωi)+ϵ) be the sum of i from 1 to m, where ϵ is 10. -4 The final output weighted sum O is ∑ωi norm ·Ii, summing i from 1 to m.

[0055] For samples with clear defects and appropriate scale, FNF may focus more on high-level semantics, while for samples with fuzzy or small defects, it enhances the contribution of detailed features. At the same time, the normalization of FNF keeps the fusion output within a controllable range, avoiding the explosion or disappearance of weight values. Combined with ReLU, it ensures positive incentives and is conducive to stable training.

[0056] As a specific implementation, in the bottom-up fusion process via downsampling, Linear Deformable Convolution (LDConv) is used for downsampling. This includes: The offset prediction convolutional layer generates 2N offsets for each spatial location, where N is the predefined number of sampling points. The offsets are added to the sampling grid coordinates to obtain the deformed sampling position, and bilinear interpolation is performed on the integer coordinate points around it to collect N feature vectors. The samples are rearranged along the height direction so that the N sampling points at the same spatial location are stacked continuously and then input into the standard convolutional layer to obtain the downsampled feature map.

[0057] LDConv learns a set of variable sampling offsets for each spatial location, so that the convolution kernel is no longer limited to a regular grid, thus better fitting irregular shapes such as slenderness and curvature in steel defects.

[0058] Specifically, firstly, an offset prediction convolutional layer generates 2N offsets for each spatial location of the input feature map, where N equals the predefined number of sampling points (e.g., N=6 for a 2×3 grid). Next, the initial regular sampling grid coordinates (the shape matches the layout of the convolutional kernel) are added to the offsets to obtain N deformed sampling locations. Bilinear interpolation is then performed at four integer coordinate points around these locations to collect N feature vectors corresponding to each location. Then, these sampled features are rearranged along the height direction, so that the N sampling points at the same spatial location are stacked continuously in the vertical direction, forming a new feature tensor. Finally, a standard convolution with a kernel size of (N, 1) and a stride of (N, 1) (combined with batch normalization and SiLU) performs channel blending and spatial projection on the stacked features, outputting the deformed feature map. This design allows the convolutional kernel to break free from the constraints of fixed geometry and adaptively focus on more critical regions. Through learnable offsets and specific feature rearrangement methods, it can effectively capture structural features with specific orientations, making it particularly suitable for handling slender strip-shaped targets commonly found in steel defect detection.

[0059] Steel defects (such as cracks and scratches) are characterized by their elongated shape, curvature, and varying width. RT-DETR's original downsampling typically uses 3×3 square convolutions (stride 2), whose fixed square receptive field is insufficient for efficiently capturing the trajectory of these defects. Replacing the downsampling with LDConv using 2×3 convolution kernels aligns perfectly with the elongated shape of steel surface cracks and scratches extending along a specific direction. This allows for the priority extraction of long-range defect trajectory information during the downsampling stage, reducing redundant sampling in the horizontal direction. Furthermore, LDConv dynamically adjusts the position of each sampling point through a learnable offset. For curved cracks or wavy scratches, the offset ensures that sampling points are distributed along the actual curvature of the defect, rather than being fixed on a regular grid.

[0060] As a specific embodiment, the steel surface defect detection method based on RT-DETR also includes model training and evaluation steps: obtaining a steel surface defect sample set and dividing it into a training set, a validation set and a test set; The defect detection model is trained iteratively using a pre-defined training set. Training is stopped and the model parameters are saved when the number of training rounds reaches the preset number of rounds. Based on the validation set, the model performance index is evaluated, and the best performing model on the validation set is determined as the optimal defect detection model.

[0061] In this embodiment, the input image was resized to 640×640. The training device was an NVIDIA RTX 4090D with 24GB of VRAM and a 16v CPU Intel(R) Xeon(R) Platinum 8481C. The environment used was PyTorch 2.3.0, CUDA 12.1, and Python 3.12.3. During training, the network model was optimized using the AdamW optimizer with an initial learning rate of 0.0001. The network was trained for a total of 300 epochs with a batch size of 8.

[0062] The model was evaluated using mean precision (mAP), accuracy (P), recall (R), and computational complexity (GFLOPs).

[0063] Mean accuracy (mAP) is a key metric for evaluating model performance, specifically including two forms: mAP50 and mAP50-95. The former uses a fixed Intersection over Union (IoU) threshold of 0.5 to measure the model's detection performance under relaxed positioning requirements; the latter calculates the average accuracy for multiple IoU thresholds ranging from 0.5 to 0.95, with a step size typically of 0.05, and then takes the overall average, thus more comprehensively characterizing the model's overall detection capability under different positioning accuracy requirements. mAP is calculated as 1 / C∑APi, where i is the summation from 1 to C, where C is the number of classes, and APi is the AP value of the i-th class.

[0064] Accuracy is expressed as P = TP / (TP + FP), where TP is the true positive (predicted positive sample correctly) and FP is the false positive (predicted positive sample incorrectly).

[0065] Recall is expressed as R = TP / (TP + FN), where TP is the true positive (predicted positive sample and correct) and FN is the false negative (true positive sample but not detected).

[0066] Computational complexity (GFLOPs) is a commonly used metric for measuring the computational complexity of a model, measured in billions of floating-point operations. It represents the total number of floating-point operations required for the model to complete one forward inference iteration. Table 1 shows a comparison of the defect detection performance of the WCL-DETR defect detection model of this invention with other existing detection models.

[0067] Table 1

[0068] Compared to baseline models, the WCL-DETR defect detection model of this invention achieves improved accuracy while reducing computational cost by approximately 16%, and also outperforms the baseline model in both accuracy and recall. Compared to the YOLO series, the WCL-DETR defect detection model of this invention leads in accuracy while using the least computational power. Compared to the two-stage Faster R-CNN model, it achieves improved accuracy while requiring only 35.5% of the computational power of Faster R-CNN. Therefore, the defect detection model of this invention can ensure higher detection accuracy with less computational power compared to existing technologies.

[0069] The above embodiments are further elaborations and descriptions of the present invention to facilitate understanding, and are not intended to limit the present invention in any way. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting surface defects in steel based on RT-DETR, characterized in that, include: A defect detection model is constructed using the RT-DETR model as the baseline model. The steel surface image is input into the backbone network, features P3, P4 and P5 are extracted, and feature Y5 is obtained by enhancing P5. P3, P4, and Y5 are input into the pyramid context extraction module to obtain features P3', P4', and Y5'. P3, P4, and Y5 are input into the rectangular self-calibration module to obtain features P3”, P4”, and Y5”. Y5” and Y5’ are first fused together to obtain M5; P4” and M5 are second fused together, and then P4’ is first fused together to obtain M4; After P3” and M4 undergo a second fusion, they are then fused together with P3’ to obtain M3; M3 is downsampled and then fused with M4 to obtain N4. N4 is downsampled and then fused with N5 to obtain N5. M3, N4, and N5 are fed into the detection head for defect detection.

2. The method for detecting surface defects in steel based on RT-DETR according to claim 1, characterized in that, The process of enhancing P5 to obtain feature Y5 employs the AIFI-WPA intra-scale feature interaction module based on dual-domain attention, including: The frequency domain branch extracts high and low frequency features through wavelet transform decomposition, and after weighted fusion by MSCM, multi-head self-attention is performed to obtain the first output; The prototype branch generates prototype features through depthwise separable convolution and dynamic clustering, and then obtains the second output after weighting by MSCM; Add the outputs of the two branches to get Y5.

3. The method for detecting surface defects in steel based on RT-DETR according to claim 2, characterized in that, The frequency domain branching process includes: The input features are decomposed using Haar wavelet decomposition to obtain low-frequency and high-frequency sub-bands, and then flattened to obtain high- and low-frequency features. Weights are generated through a multi-scale context module, and high- and low-frequency features are then fused using weighted methods. The fusion result is used as the key and value, and the original query Q is used as the query. Multi-head self-attention is performed, and the first output is obtained through residual connection and layer normalization.

4. The method for detecting surface defects in steel based on RT-DETR according to claim 2, characterized in that, The processing of the prototype branch includes: The input features are subjected to depthwise separable convolution and flattened into a sequence; The sequence generates a multi-prototype weight matrix through linear mapping, and calculates the prototype and interpolates the source features of the sub-band sum after wavelet decomposition. The gating weights are obtained through the multi-scale context module, and the residual weighting is performed to obtain the second output.

5. The method for detecting surface defects in steel based on RT-DETR according to claim 1, characterized in that, The processing procedure of the pyramid context extraction module is as follows: The input features P3, P4, and Y5 are adaptively pooled to a uniform minimum size to obtain pooled features. All pooling features are concatenated along the channel dimension to obtain the aggregated feature Z; Z is then processed through several rectangular self-calibrating attention modules. The processed features are split according to the original number of input channels to obtain enhanced features P3', P4', and Y5'.

6. A method for detecting surface defects in steel based on RT-DETR according to claim 1 or 5, characterized in that, The processing procedure of the rectangular self-calibration module is as follows: The input features are processed through a rectangular self-calibrating attention mechanism to extract axial global context information and then normalized. The normalized features are input into a convolutional multilayer perceptron with GELU activation and then multiplied by the layer scaling parameter. The scaled features are added to the original input features to obtain the output features.

7. A method for detecting surface defects in steel based on RT-DETR according to any one of claims 1-5, characterized in that, The first fusion is a cross-scale gated fusion, which generates spatial attention weights by performing 1×1 convolution and h-sigmoid activation on high-level features, and then modulates the low-level features element by element after upsampling. The second fusion is a bilinear additive fusion, which involves adjusting the channels of the upsampled high-level features and adding them element-by-element to the low-level features.

8. A method for detecting surface defects in steel based on RT-DETR according to any one of claims 1-5, characterized in that, The downsampling includes: The offset prediction convolutional layer generates 2N offsets for each spatial location, where N is the predefined number of sampling points; The sampling grid coordinates are added with an offset to obtain the deformed sampling position, and bilinear interpolation is performed at the integer coordinate points around it to collect N feature vectors; rearrange them along the height direction so that the N sampling points at the same spatial position are stacked continuously, and then input into a standard convolutional layer to obtain the downsampled feature map.

9. The method for detecting surface defects in steel based on RT-DETR according to claim 8, characterized in that, The third fusion is a fast normalization fusion, which assigns learnable weights to each input feature, obtains the corresponding fusion coefficients after ReLU and normalization, and then performs a weighted summation.

10. A method for detecting surface defects in steel based on RT-DETR according to any one of claims 1-5, characterized in that, It also includes model training and evaluation steps: Obtain a sample set of surface defects in steel and divide it into a training set, a validation set, and a test set; The defect detection model is trained iteratively using a pre-defined training set. Training is stopped and the model parameters are saved when the number of training rounds reaches the preset number of rounds. Based on the validation set, the model performance index is evaluated, and the best performing model on the validation set is determined as the optimal defect detection model.

Images