Space-frequency domain coupled industrial surface defect detection method and system based on improved YOLO

Abstract

This invention discloses a spatial-frequency domain coupled industrial surface defect detection method based on an improved YOLO model, comprising: Step 1: acquiring defect images under the same working conditions and completing annotation and data segmentation; Step 2: performing data augmentation processing on the sample images to improve the robustness and generalization ability of the model; Step 3: adopting spatial wavelet blocks, wavelet-sensory multi-scale spatial modulation, dual-path token fusion module, and token wavelet pyramid network to innovate the structure of the YOLOv5 model, and introducing local density and geometric adaptive nonmaximum suppression to optimize overlapping box filtering; Step 4: performing structured pruning and knowledge distillation on the trained model to obtain a lightweight model, improving deployment efficiency in industrial settings while maintaining detection accuracy; Step 5: deploying the model in industrial settings to achieve real-time inference and defect localization output. This invention exhibits higher detection rate and performance in scenarios with small targets, multiple scales, and extremely large aspect ratio defects.

Description

Technical Field

[0001] This invention belongs to the field of industrial visual inspection and intelligent manufacturing technology, specifically relating to a spatial-frequency domain coupled industrial surface defect detection method and system based on improved YOLO. Background Technology

[0002] Industrial surface defect detection is crucial for improving product quality consistency and reliability. Timely and accurate defect identification directly impacts product yield and manufacturing costs, and provides a basis for process control, equipment maintenance, and quality traceability. Existing detection solutions often rely on manual experience and feature design, which struggles to stably characterize defects under complex textures, imaging fluctuations, and noise interference, limiting their adaptability and transferability.

[0003] In recent years, object detection methods based on convolutional neural networks have been widely used in industrial defect detection due to their end-to-end feature learning capabilities. However, they still face several key challenges in actual working conditions: On the one hand, small defects have small areas and weak contrast, making them easy to be overwhelmed by repetitive textures and random noise, resulting in missed detections; on the other hand, defects with extremely large aspect ratios have obvious directionality and uneven coverage, leading to unstable localization box regression and category discrimination; in addition, defects on the same production line have large scale spans and significant cross-scale distributions, making it difficult for single-scale representation and static fusion strategies to simultaneously ensure detail fidelity and global semantic consistency, easily resulting in both false detections and missed detections.

[0004] To improve detection performance in complex scenarios, existing technologies typically introduce attention mechanisms, multi-scale pyramids, and deformable modeling from the spatial domain to enhance representation capabilities. However, these methods still fall short in handling challenging cases such as backgrounds with strong textures, extreme shapes, and large scale spans. Meanwhile, frequency domain analysis methods can separate scale and rate of change components across frequency bands to suppress background interference and highlight defect signals; however, without co-modeling with spatial information, spatial localization information can be weakened. Furthermore, cross-layer features are prone to detail attenuation and semantic mismatch during resampling and fusion, affecting the alignment and focusing of defect regions.

[0005] Furthermore, the non-maximum suppression strategy in the post-processing stage also significantly affects the detection results. Existing suppression methods based on fixed thresholds lack adaptive adjustment capabilities for target density, scale, and geometry. In scenarios with dense defects, adjacent defects, or significant morphological differences, they are prone to over-suppression leading to missed detections or under-suppression leading to false alarms. For these reasons, there is an urgent need for an industrial surface defect detection solution that balances accuracy, stability, and deployability under complex operating conditions to collaboratively address typical challenges such as small targets, extreme aspect ratios, and cross-scale distributions. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a spatial-frequency domain coupled industrial surface defect detection method and system based on an improved YOLO, which enhances detection performance through key technologies such as spatial-frequency collaborative characterization, cross-layer alignment fusion, adaptive post-processing, and model compression deployment. Specifically, spatial wavelet blocks are used to achieve parallel modeling and fusion of spatial and frequency domain branches. While maintaining global texture and semantic consistency, high-frequency details are supplemented and the localization alignment capability is improved, thereby enhancing the distinguishable representation of small defects. A wavelet-aware multi-scale spatial modulation mechanism is introduced to guide channel calibration with low-frequency information and drive directional gain with high-frequency information. Combined with multi-scale convolution, channel, multi-scale space and orientation are jointly adjusted to improve the discrimination capability of cross-scale and high aspect ratio defects. A dual-path token fusion module is used and a token wavelet pyramid network is constructed to achieve cross-layer feature alignment and selective aggregation, reduce detail attenuation caused by resampling and enhance the focusing effect on defect regions. In the post-processing stage, a local density and geometric adaptive non-maximum suppression strategy is introduced to make the candidate box suppression intensity adaptively change with the target density and geometry and compensate for the area and aspect ratio, thereby reducing false alarms and missed detections caused by over-suppression and under-suppression. Furthermore, this invention introduces sparsity constraints into the training model to highlight redundant structures, and performs structured pruning to reduce the number of parameters and computational load. Subsequently, knowledge distillation transfers the discriminative ability of the teacher model to the lightweight model, aiming to maintain detection accuracy while improving inference speed and reducing hardware resource consumption. Finally, the lightweight model is integrated into mobile or industrial edge devices to achieve real-time inference and localization output of industrial surface defects. The optimal model is selected for deployment through comparative evaluation of candidate models, thereby verifying the engineering usability and effectiveness of this invention.

[0007] To achieve the above objectives, the technical solution adopted by this invention is as follows: A multi-scale feature fusion detection method for industrial surface defects based on improved YOLO, characterized by comprising the following steps: To achieve the above objectives, the technical solution adopted by this invention is as follows: A spatial-frequency domain coupled industrial surface defect detection method based on improved YOLO, comprising the following steps:

[0008] Step 1: Collect defect images under the same production conditions and complete the annotation and dataset division;

[0009] Step 2: Perform data augmentation on the sample images to improve the robustness and generalization ability of the model;

[0010] Step 3: The YOLOv5 model is structurally innovated by using spatial wavelet blocks, wavelet sensing multi-scale spatial modulation, dual-path token fusion module and token wavelet pyramid network, and local density and geometric adaptive nonmaximum suppression are introduced to optimize the selection of overlapping boxes.

[0011] Step 4: Perform structured pruning and knowledge distillation on the trained model to obtain a lightweight model, improving deployment efficiency in industrial applications while maintaining detection accuracy;

[0012] Step 5: Deploy the model in industrial settings to achieve real-time inference and defect location output.

[0013] Furthermore, in step one, the method for obtaining sample image data includes:

[0014] The LabelImg annotation tool was used to manually annotate the collected industrial surface defect images. For each defect, its category information and spatial location were marked with a minimum bounding rectangle. The boundary of the minimum bounding rectangle was used to completely cover the corresponding defect area to ensure that the defect target was not truncated and the background redundancy was minimized, so as to form annotated sample image data for training and validation.

[0015] Furthermore, in step two, the methods for data augmentation of the sample image data include:

[0016] Spatial and imaging perturbations are applied to the sample images according to a preset probability to generate augmented samples. The spatial perturbation includes at least one or more of random rotation, scaling, and random cropping, and can be further superimposed with translation or flipping operations. The imaging perturbation includes at least brightness and contrast changes, and can be further superimposed with noise injection or blurring. At the same time, during the process of enhancing and generating new images, the coordinates of the defect annotation box and the defect category are synchronously mapped according to the same transformation rule to maintain consistent annotation, thereby increasing data diversity and improving the model's stable detection capability under different imaging and texture conditions.

[0017] Furthermore, the specific methods in step three include:

[0018] Spatial wavelet blocks are used to model the input features in parallel with spatial and frequency domain branches and then fuse them. The spatial branch is used to maintain global texture and semantic consistency, while the frequency branch is used to provide high-frequency detail compensation and localization alignment, thereby enhancing the distinguishable representation of minute defects.

[0019] The fused features are jointly adjusted using wavelet sensing multi-scale spatial modulation. Low-frequency information guides channel calibration and high-frequency information drives directional gain. Multi-scale convolution is used to achieve synergistic enhancement of channels, multi-scale space and direction, so as to improve the discrimination ability of cross-scale and high aspect ratio defects.

[0020] The dual-path token fusion module is used to achieve cross-layer feature alignment and selective aggregation, and together with the spatial wavelet block, a token wavelet pyramid network is constructed to progressively fuse multi-layer features under the condition of consistent scale, thereby alleviating the loss of multi-scale defect detail information and enhancing the ability to focus on defect areas.

[0021] In the prediction output stage, local density and geometric adaptive nonmaximum suppression are introduced to make the suppression intensity of the candidate box adaptively change with the target distribution density and geometry. The area and aspect ratio are compensated and adjusted to reduce over-suppression or under-suppression and stably retain the effective detection results of small defects, extreme aspect ratio defects and multi-scale defects.

[0022] Furthermore, in step four, the method for obtaining a lightweight model through pruning and distillation includes:

[0023] The improved YOLO model, after training, is subjected to sparsity constraint training to highlight the scalability of redundant structures. The sparsity constraint can be applied to the channel scaling factor and employs... Regular terms, where Defined as:

[0024] ;

[0025] in, Scaling factor for each channel, The sparsity strength coefficient is then set, followed by the calculation of the compressibility coefficient. To control the pruning amplitude, the aforementioned Defined as:

[0026] ;

[0027] in, The computational load of the model before pruning, To reduce the computational load of the model after pruning, adjust... This approach allows for controllable adjustment of the network slimming ratio. After pruning, the model undergoes structural reconstruction and retraining to restore accuracy. Knowledge distillation is performed using the unpruned model as the teacher model and the post-pruning model as the student model. The distillation loss is defined as:

[0028] ;

[0029] in, , The predicted outputs for teachers and students are respectively. The temperature coefficient is the final training objective.

[0030] ;

[0031] in, To detect mission losses, The distillation weighting coefficients are used to reduce the number of model parameters and computational load while maintaining defect detection accuracy and improving deployment efficiency as much as possible.

[0032] Furthermore, it also includes the following steps:

[0033] Multiple sets of repeated training or different hyperparameter combinations are performed on the sample image data under a preset training strategy to obtain multiple candidate training models. The training strategy includes at least different learning rates, batch sizes, input resolutions, or training rounds.

[0034] The multiple candidate training models are subjected to unified testing and performance evaluation. Based on the detection accuracy index, inference latency or frame rate index, and spatial-frequency domain coupling feature expression effect index, a comprehensive comparison and screening is performed, and the candidate model with the best comprehensive index is selected as the final deployment training model.

[0035] Furthermore, methods for performance evaluation and comparison of multiple trained models include:

[0036] A quantitative test of each candidate model is conducted using an evaluation system for object detection tasks. The evaluation system includes at least precision, recall, mean precision, and mean mean precision to characterize the detection effect. Furthermore, the computational overhead and operational efficiency parameters of the model are statistically analyzed. These parameters include at least floating-point operations, parameter size, inference throughput, and single-frame inference latency. This allows for a comprehensive evaluation and ranking of the model's detection performance and the feasibility of its deployment on the edge.

[0037] Furthermore, in step five, the defect detection method includes:

[0038] The final training model obtained in step four is converted and integrated into the object detection application. The integration includes replacing the model file in the original detection program with the weight file and network configuration file of the final training model and completing version management.

[0039] The model's input size, output tensor dimension, and post-processing parameters are checked for consistency. If necessary, the input size, category mapping relationship, and label configuration file are adjusted according to the actual camera resolution and category set to ensure that the detection category is consistent with the display result.

[0040] The integrated detection program is packaged into an installable application and deployed to the target terminal device, which includes a mobile terminal or an industrial edge computing terminal. The application is launched to call the final trained model to perform online inference and output defect category, confidence level and location box information to achieve real-time detection of industrial surface defects.

[0041] A space-frequency domain coupled industrial surface defect detection system based on an improved YOLO, comprising:

[0042] The sample acquisition module is used to acquire images of industrial surface defects.

[0043] The data annotation and segmentation module is used to annotate defect categories and locations and complete dataset segmentation;

[0044] The data augmentation module is used to augment and expand samples and update annotation information synchronously.

[0045] The model structure optimization module is used to introduce spatial wavelet blocks, wavelet-sensory multi-scale spatial modulation, dual-path token fusion module, token wavelet pyramid network and local density and geometric adaptive nonmaximum suppression on the basis of YOLOv5 to build an improved model.

[0046] The model training module is used to train the improved model using the augmented samples;

[0047] The model selection module is used to evaluate and compare multiple candidate models obtained from training and select the optimal model.

[0048] The pruning and distillation module is used to perform structured pruning and knowledge distillation on the optimal model to obtain a lightweight model;

[0049] The deployment and inference module is used to deploy the lightweight model to the target terminal and perform real-time inference to output defect detection results.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. This invention introduces spatial wavelet blocks to achieve collaborative modeling of spatial features and frequency domain details. While maintaining global texture and semantic consistency, it supplements high-frequency edge and fine-grained structural information, thereby improving the distinguishable characterization and stable detection capability of minute defects against a strong texture background.

[0052] 2. This invention utilizes a wavelet sensing multi-scale spatial modulation mechanism to apply low-frequency information for channel calibration and use high-frequency information to drive directional gain. Combined with multi-scale convolution, it achieves joint enhancement of channel, space, and direction, thereby improving the ability to locate and distinguish cross-scale distributed defects and extreme aspect ratio defects.

[0053] 3. This invention employs a dual-path token fusion module and constructs a token wavelet pyramid network to achieve cross-layer feature alignment and selective aggregation, reduce detail attenuation and semantic mismatch caused by resampling, enhance the focusing effect on defective regions, and reduce missed detections and false detections.

[0054] 4. In the post-processing stage, this invention introduces a local density and geometric adaptive non-maximum suppression strategy, which makes the suppression intensity adaptively change with the target density and geometry and compensates for the area and aspect ratio, thereby improving the candidate box screening effect in scenarios with dense defects, adjacent defects and significant morphological differences, and improving the stability and consistency of the detection results.

[0055] 5. This invention applies sparsity constraints to the training model and performs structured pruning. Combined with knowledge distillation, it transfers the capabilities of the teacher model to the lightweight model. While significantly reducing the number of parameters and computational load and improving inference speed, it maintains detection accuracy as much as possible. This makes it easy to deploy in real time on mobile devices or industrial edge devices, thereby reducing hardware costs and improving the engineering usability of industrial online detection. Attached Figure Description

[0056] Figure 1 This is a flowchart of the spatial frequency domain coupling industrial surface defect detection method of the present invention;

[0057] Figure 2 This is a schematic diagram of the overall structure of the improved YOLOv5 detection model of this invention;

[0058] Figure 3 This is a schematic diagram of the spatial wavelet block structure of the present invention;

[0059] Figure 4 This is a schematic diagram of the wavelet sensing multi-scale spatial modulation module of the present invention;

[0060] Figure 5 This is a schematic diagram of the dual-path token fusion module of the present invention;

[0061] Figure 6 This is a schematic diagram illustrating the model compression principle of the present invention, which incorporates sparsity constraints, structured pruning, and knowledge distillation.

[0062] Figure 7 This is a schematic diagram of the industrial surface defect detection system of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some and not all embodiments of the present invention; any substitutions, modifications or combinations made by those skilled in the art based on the concept of the present invention without creative effort should fall within the protection scope of the present invention.

[0065] Figure 1 This is a schematic flowchart of the method of the present invention. An embodiment of the present invention provides a spatial-frequency domain coupled industrial surface defect detection method based on an improved YOLO, comprising the following steps:

[0066] Step 1: Collect images of industrial surface defects under the same production conditions, and complete the annotation and dataset division according to the defect category to construct sample image data;

[0067] Step 2: Perform data augmentation on the sample image data by expanding the sample distribution through spatial and imaging perturbations and updating the annotation information simultaneously;

[0068] Step 3: Based on YOLOv5, structural improvements are made by using spatial wavelet blocks, wavelet sensing multi-scale spatial modulation, dual-path token fusion module and token wavelet pyramid network to achieve spatial-frequency domain collaborative representation and cross-layer alignment fusion, and local density and geometric adaptive nonmaximum suppression are introduced to optimize candidate box screening.

[0069] Step 4: Apply sparsity constraints to the training model and perform structured pruning, while combining knowledge distillation to obtain a lightweight model, so as to reduce the amount of computation and maintain detection accuracy as much as possible;

[0070] Step 5: Integrate the lightweight model into mobile devices or industrial edge devices to achieve real-time detection and location output of industrial surface defects.

[0071] The specific steps are as follows:

[0072] S1. Collect industrial surface images under target production line or equivalent working conditions to form raw sample data;

[0073] S2. Use annotation tools to annotate the defect categories and locations, and divide the training set, validation set and test set according to a preset ratio to obtain sample image data;

[0074] S3. Perform data augmentation on the sample image data, the augmentation including geometric perturbation and imaging perturbation, and synchronously map and update the coordinates of the bounding boxes during the augmentation generation process to maintain label consistency;

[0075] S4. An improved detection network is built on the basis of YOLOv5, and spatial wavelet blocks are introduced to realize parallel modeling and fusion of spatial and frequency domain branches, so as to enhance the distinguishable characterization of small defects;

[0076] S5. Introduce a wavelet sensing multi-scale spatial modulation mechanism, use low-frequency information for channel calibration and high-frequency information to drive directional gain, and combine multi-scale convolution to achieve joint enhancement of channel-space-direction, so as to improve the ability to distinguish cross-scale and high aspect ratio defects.

[0077] S6. A dual-path token fusion module is adopted and a token wavelet pyramid network is constructed to achieve cross-layer feature alignment and selective aggregation, reduce the detail attenuation caused by resampling and enhance the focusing effect on defective regions. At the same time, local density and geometric adaptive non-maximum suppression are introduced in the prediction output stage to optimize candidate box selection.

[0078] S7. In the prediction output stage, local density and geometric adaptive nonmaximum suppression are introduced. The suppression intensity of the candidate box is adaptively adjusted according to the target density and geometry, and area and aspect ratio compensation are performed to obtain the final detection result.

[0079] S8. A lightweight model is obtained by applying sparsity constraints, structured pruning, and knowledge distillation to the trained model;

[0080] S9. Deploy the spatial frequency domain coupling model onto the target mobile device for industrial surface defect detection.

[0081] In step S1, the method for acquiring images of industrial surface defects is as follows:

[0082] S1A: Image acquisition is carried out under the same production conditions, covering multiple defect types and different defect shapes, scales and textures to ensure that the samples are representative and meet the needs of subsequent training.

[0083] S1B: The resolution of the acquired images is set to 1920×1080, and the images are saved in PNG format and uniformly named and archived.

[0084] In step S2, the method for annotating and dividing the collected industrial surface defect images to obtain sample image data is as follows:

[0085] S2A: The LabelImg annotation tool is used to select and annotate the defect targets in each image. The minimum bounding rectangle is used to completely cover the defect area and record the defect category and the coordinate information of the bounding box. The annotation information is represented as (class, xmin, ymin, xmax, ymax), where class is the defect category, xmin and ymin are the coordinates of the upper left corner of the annotation box, and xmax and ymax are the coordinates of the lower right corner of the annotation box, so as to achieve a precise description of the defect type and spatial location.

[0086] S2B: All labeled samples are divided into training set, validation set and test set according to the ratio of 8:1:1 and stored separately. Image files are stored in the images directory and corresponding labeled files are stored in the labels directory. A one-to-one correspondence is established between each image file and the labeled file through the same name rule.

[0087] In step S3, the method for data augmentation of the sample image data is as follows:

[0088] S3A: Addressing the characteristics of industrial surface images, such as significant differences in defect scale, complex textures and backgrounds, and fluctuating imaging conditions, an enhancement strategy combining geometric and imaging perturbations is applied to training samples according to preset probabilities. The geometric perturbations include random rotation, scale transformation, and random cropping, with optional overlay translation or flipping to expand the distribution of defects in different poses and sizes. The imaging perturbations include brightness and contrast perturbations, with optional overlay noise injection or blurring to improve the model's adaptability to changes in illumination and acquisition noise. Optionally, stitching enhancement can be used to combine multiple images to increase scene diversity. During the above enhancement operations, the coordinates of the defect annotation boxes are simultaneously and consistently mapped while maintaining the defect category, thereby generating expanded samples for training. The improved YOLOv5 structure is as follows: Figure 2 As shown.

[0089] like Figure 3 This reflects the principle of spatial wavelet blocks.

[0090] In step S4, spatial wavelet blocks are used to extract industrial defect features. Parallel modeling in the spatial and frequency domains is used to achieve alignment and fusion at the same scale. High-frequency detail compensation and directional information are introduced while maintaining spatial positioning capabilities, thereby enhancing the characterization ability of small defects and defects with large aspect ratios. Specifically, this includes:

[0091] S4A: The input feature map is first split into two parallel branches: a frequency domain enhancement branch and a spatial domain enhancement branch. The frequency domain enhancement branch is used to extract and enhance low-frequency global and high-frequency detail information, while the spatial domain enhancement branch is used to supplement the local spatial response and maintain the continuity of the original spatial structure.

[0092] S4B: Specifically, SWB Conv takes the input feature map as... Starting from this point, the spatial and frequency domain branches are executed in parallel and then fused for output. The frequency domain branch first uses first-order Haar wavelet decomposition and reconstruction to calculate four orthogonal sub-bands. Then, downsampling is performed with a step size of 2 to obtain the four sub-bands. The above process can be described as follows:

[0093]

[0094]

[0095]

[0096]

[0097] in, and These are wavelet reconstruction low-pass and high-pass filters, respectively. For the outer product, It is a two-dimensional convolution. This is a 2x downsampling.

[0098] The four subbands are then stacked and reshaped, a DW Conv layer is applied, and then inverse reshaping is performed to obtain the enhanced four subbands, which are then aligned with spatial features, thereby effectively suppressing noise and supplementing detailed features. The above process can be described as follows:

[0099]

[0100]

[0101]

[0102] ;

[0103] Where Stack represents the stacking operation. and These are reshaping and deshaping operations, respectively.

[0104] Finally, a two-dimensional separable synthetic core was utilized. Perform an inverse wavelet reconstruction with an upsampling step of 2 to obtain frequency domain enhancement features with the same resolution as the input. The above process can be described as follows:

[0105] ;

[0106] in, This is a 2x upsampling.

[0107] S4C: The spatial branch uses two layers of DW Conv to supplement the local feature responses. The first layer has an inflation rate of 1, and the second layer has an inflation rate of 2. The outputs of the two layers are multiplied by a learnable coefficient. and The summation of the two elements creates a spatially enhanced feature. The above process can be described as follows:

[0108] ;

[0109] in, and For learnable scaling factor, , .

[0110] S4D: In the fusion stage, we directly add the two branches while maintaining pixel alignment, outputting a feature map. This allows for the complementarity of spatial and frequency domain information on the same scale. The above process can be described as follows:

[0111]

[0112] like Figure 4 This reflects the principle of wavelet sensing multi-scale spatial modulation module.

[0113] In step S5, the feature map output from step S4 is jointly modulated using a wavelet-sensing multi-scale spatial modulation module. This module consists of three parallel branches: a multi-scale depth-separable convolution mechanism, low-frequency guided channel attention, and high-frequency driven directional attention. It enhances the discrimination capability against cross-scale and high aspect ratio defects through multi-scale structural enhancement, channel calibration, and directional reinforcement. The method is as follows:

[0114] S5A: Let the feature map output by step S4 be... ,Will Parallel input is fed into three branches, and branch outputs are obtained from each branch. , and The output feature map is obtained by fusing the features at the same scale. .

[0115] S5B: A branch of multi-scale depthwise separable convolution mechanism in The operation is performed in two paths. The first path uses depthwise separable convolutions (DW Conv) with different kernel sizes to generate weights. The second path fusion generates weights. Finally passed Weighted summation This branch is used to capture local structural differences across different receptive fields, enhancing robustness to defects with large size ranges. The above process can be described as follows:

[0116]

[0117]

[0118] ;

[0119] In the formula is the ReLU activation function, and BatchNorm is batch normalization.

[0120] S5C: For the low-frequency guiding channel attention and high-frequency driving direction attention branches, we first analyze the feature map. Haar wavelet decomposition is performed to obtain four subbands. The above process can be described as follows:

[0121]

[0122]

[0123] ;

[0124] in, For the outer product, It is a two-dimensional convolution. This is a 2x downsampling.

[0125] S5D: In the low-frequency lead channel attention branch, we upsample the low-frequency subband to... The same spatial dimensions. Then, the operation is performed in two paths. The first path obtains the channel weights. The second path retains the original features. Output the results. Finally, assign the channel weights. With original features Multiply each channel sequentially to obtain the output of that branch. This branch, guided by low-frequency global information, enhances the channel dimension, effectively suppressing large-area background undulations and highlighting attribute channels related to defects. The above process can be described as follows:

[0126]

[0127] ;

[0128] In the formula For upsampling, It is the Sigmoid activation function. This is a one-way multiplication.

[0129] S5E: In the high-frequency drive direction attention branch, we upsample the high-frequency subbands to the same level as... The same spatial dimensions are used. Then, the operation is performed in two paths. The first path calculates the mean along the channel dimension and extracts the horizontal, vertical, and diagonal texture responses through three types of directional convolutions to obtain a spatial attention map. The second path also preserves the original features. Produce the output. Finally, pay attention to the space. With original features Multiply each channel sequentially to obtain the output of that branch. This branch utilizes the directional information of high-frequency edges and fine-line textures to significantly improve the response and positioning stability to defects with extremely large aspect ratios. The above process can be described as follows:

[0130]

[0131]

[0132]

[0133] ;

[0134] S5F: Finally, the weighted sum of the three-branch outputs yields the fused feature, which is then multiplied by the learnable scaling factor. Add the residuals together to obtain the final output. This achieves the integration of multi-scale, channel, and directional information, further enhancing the module's ability to express multi-scale and directional information. The above process can be described as follows:

[0135]

[0136]

[0137]

[0138] In the formula This is the SiLU activation function.

[0139] like Figure 5 This reflects the principle of the dual-path token fusion module in the token wavelet pyramid network of the neck region.

[0140] In step S6, a token wavelet pyramid network is used to replace the neck structure of the YOLOv5 model. While maintaining the bidirectional flow structure of the feature pyramid network, two key improvements are made. First, after each feature fusion, wavelet subband detail compensation is performed using spatial wavelet blocks. Second, a dual-path token fusion module is designed at three different scales. Through two paths—directional and global gating—it generates a fusion mask that adapts to position and scale, improving cross-layer alignment and selective aggregation capabilities. This feature flow path enhances high-frequency detail fidelity and multi-scale alignment, reducing distortion and mismatch in cross-scale fusion. The principle of the dual-path token fusion module is as follows:

[0141] S6A: In the dual-path token fusion module, the input feature map is first processed... Perform projection and activation, then encode into a token sequence. Furthermore, through fully connected layers and residuals, cross-spatial interactions are established at various locations, while stochastic depth regularization is used to suppress overfitting. The above process can be described as follows:

[0142]

[0143] ;

[0144] S6B: Next, branch along the local direction. After transposing to a spatial shape, average pooling and concatenation are performed in both the horizontal and vertical directions, and then the result is passed through a fully connected layer to obtain the gated vector. This effectively enhances directional features such as high aspect ratio defects. Then, along the global feature branch... By averaging along the token dimension, a global description is generated, and similarly, gating is obtained. This is used to capture the overall texture and contextual information of defects and suppress local noise. The results from the two branches are then averaged and fused, and finally decoded using tokens. The above process can be described as follows:

[0145]

[0146] ;

[0147] In the formula, This is a concatenated vector obtained by performing average pooling in both the horizontal and vertical directions. The global feature vector is obtained by averaging all tokens along the feature dimension. , , and It is a fully connected layer for MLP.

[0148] S6C: Finally, it undergoes adaptive average pooling and learnable factor operations. This allows the dual-path token fusion module to retain original information while also considering local, global, and channel features, significantly improving the detection accuracy for small-scale and linear defects and effectively mitigating the problem of detail loss when dealing with multi-scale defects. The above process can be described as follows:

[0149] ,

[0150] ;

[0151] In step S7, local density and geometric adaptive nonmaximum suppression is used to post-process the candidate boxes of the improved YOLO model. This allows the suppression intensity to be adaptively adjusted according to the distribution density and geometry of the candidate boxes, and compensation is made for the area and aspect ratio biases. This effectively suppresses redundant boxes while more robustly preserving small defects, extreme aspect ratio defects, and cross-scale defects. The method is as follows:

[0152] S7A: First, a fractional augmentation with local density adaptive nonmaximum suppression is designed, based on local density and geometry adaptive nonmaximum suppression. For local density, it directly depends on the optimal bounding box. The IoU distribution with its surrounding candidate boxes. When When the surrounding candidate boxes of the same type are sparse Value close to This allows for additional enhancement of remaining smaller objectives, preventing them from being falsely suppressed due to naturally low overlap. Conversely, when When the surrounding areas are densely overlapping, Approaching 1, A value close to 1 will not interfere with normal suppression. For adaptive Gaussian decay, it will adjust the parameter... Set as For high IoU boxes, Smaller size results in more dramatic attenuation, improving the efficiency of eliminating high-overlap boxes. For low-IoU boxes, Larger, with gradual attenuation, in combination This can further protect smaller targets. The above process can be described as follows:

[0153]

[0154] S7B: Then, geometrically consistent modulation was designed using local density and geometrically adaptive nonmaximum suppression. For area compensation, when This weakens the suppression of small targets in the Gaussian decay term. And when... When large targets are not compensated, no additional compensation is given. For shape adaptation, shape similarity is evaluated using aspect ratio differences. When candidate boxes... With best box For shapes that are similar, the score remains almost unchanged. When the shape difference is significant, the score is further reduced to decrease false retention of shape-mismatched bounding boxes. For elongated, flat, or other special shape defects, false suppression under IoU-dominated conditions is reduced, improving the detection accuracy of these targets. The above process can be described as follows:

[0155]

[0156]

[0157] S7C: Local density and geometrically adaptive nonmaximum suppression, unified weights and updates can be expressed as:

[0158]

[0159]

[0160] S8: Select the experimental environment configuration as shown in Table 1, set the training hyperparameters as shown in Table 2, and start training the improved model after setting the parameters.

[0161] Table 1 Experimental Environment Configuration

[0162]

[0163] Table 2 Experimental hyperparameter settings

[0164]

[0165] In step S8, the trained improved YOLO model undergoes structured pruning and is combined with distillation to obtain a lightweight model, as follows:

[0166] S8A: To achieve controllable slimming through structured pruning, this invention first performs sparsity constraint training on the trained model, and introduces a scaling factor into the batch normalized layer channel scaling factor of the model. The regularization term makes the scaling factor of some channels approach 0, thus explicitly distinguishing the importance of channels and providing a basis for subsequent pruning. Its sparsity regularization term is defined as:

[0167] ;

[0168] in, Scaling factor for each channel, is the sparsity intensity coefficient.

[0169] S8B: Then set the calculation of the compression factor. To control the pruning amplitude, the aforementioned This is used to characterize the degree of compression of model computation before and after pruning, and as a control quantity for pruning intensity. By constraining the pruning process to ensure the model meets the target computing power and latency requirements, a controllable trade-off is achieved between improving inference speed and maintaining detection accuracy. Defined as:

[0170] ;

[0171] in, The computational load of the model before pruning, This represents the computational cost of the model after pruning.

[0172] S8C: Knowledge distillation is performed using the pre-pruning model as the teacher model and the post-pruning model as the student model. This knowledge distillation is used to transfer the implicit inter-category relationships, hard sample discrimination boundaries, and feature response distributions from the teacher model to the student model after the model structure has been pruned and compressed. This compensates for the decreased expressive power caused by pruning and steadily improves the detection accuracy and recall of the student model in scenarios involving small defects, cross-scale defects, and extreme aspect ratio defects. Simultaneously, it reduces convergence oscillations during retraining and improves the generalization stability of the lightweight model. The distillation loss is defined as:

[0173] ;

[0174] in, , The predicted outputs for teachers and students are respectively. This is the temperature coefficient.

[0175] S8D: The ultimate training objective is to jointly optimize the detection task loss and knowledge distillation loss, so that the student model, while meeting the constraints of lightweight and low computational cost, can approximate the prediction distribution and discrimination ability of the teacher model as closely as possible. This will maintain stable localization accuracy and classification consistency after pruning and compression, and improve generalization performance and inference stability under different texture backgrounds and imaging perturbation conditions. The ultimate training objective is defined as follows:

[0176] ;

[0177] in, To detect mission losses, The distillation weighting coefficients are used to reduce the number of model parameters and computational load while maintaining defect detection accuracy and improving deployment efficiency as much as possible.

[0178] The spatial-frequency domain coupling method for industrial surface defect detection based on the improved YOLO also includes the following steps:

[0179] S8': Repeatedly train the improved YOLO model using sample image data or different hyperparameter combinations to obtain multiple candidate training models; perform unified testing and performance comparison on these multiple candidate training models, and select the candidate model with the best overall performance as the final training model based on indicators such as detection accuracy, computational complexity, and inference efficiency. The specific method is as follows:

[0180] S8'A: Performance evaluation metrics and comparison methods.

[0181] Each candidate trained model is evaluated on the same test set, and the metrics used include at least: accuracy. Recall rate Average accuracy Mean accuracy Floating-point arithmetic complexity Parameter quantity ,frames per second and single-frame inference latency This is to comprehensively evaluate the model's detection performance, computational scale, and operational efficiency.

[0182] S8'B: Definition of detection accuracy related indicators.

[0183] Let TP be the number of correctly detected defects, FP be the number of falsely detected defects, and FN be the number of missed defects. and It can be defined as:

[0184]

[0185]

[0186] For the first Class of defects, their average accuracy The area under the precision-recall curve is defined as:

[0187]

[0188] If the number of defect categories is N, then the mean accuracy is... Defined as:

[0189]

[0190] Where N is the number of defect categories.

[0191] S8'C: Definition of model complexity and inference efficiency metrics.

[0192] Taking a convolutional layer as an example, let the size of the output feature map be... The kernel size is The number of input channels is The number of output channels is The computational complexity of this layer can then be expressed as:

[0193]

[0194] The corresponding parameter quantity can be expressed as:

[0195]

[0196] Let the total time spent on the test set on the target device be... The average inference time for a single image is:

[0197]

[0198] And there are:

[0199]

[0200]

[0201] in, Used to reflect the end-to-end inference latency of a single frame.

[0202] S8'D: Comparison of structural schemes and determination of the optimal structure.

[0203] While maintaining consistency in training data, input size, training epochs, and optimizer strategy, comparative experiments were conducted on different structural schemes. These schemes included at least a combination of the basic YOLOv5 structure and schemes incorporating spatial wavelet blocks, wavelet-sensory multi-scale spatial modulation, a dual-path token fusion module, and a token wavelet pyramid network. The results were compared... Determine the optimal network structure configuration based on false positives or false negatives.

[0204] S8'E: Comparison of post-processing strategies and determination of the optimal strategy.

[0205] While maintaining consistency with the network structure and training strategy, different candidate box suppression strategies are compared, including traditional non-maximum suppression and the local density and geometric adaptive non-maximum suppression proposed in this invention. The comparisons are made by examining changes in recall, false deletions in dense defect scenarios, and the final... Based on performance, determine the optimal post-processing configuration.

[0206] S8'F: Comparison of compression parameters and distillation configuration.

[0207] After determining the optimal structure and post-processing strategy, different computational compression coefficients K and different distillation weight coefficients were further set. Comparative experiments were conducted, and the lightweight model with the best comprehensive performance under the constraints of computing power and latency of the target device was selected as the final deployment model.

[0208] In all the comparative experiments mentioned above, consistent experimental conditions should be maintained, including but not limited to the same dataset partition ratio (8:1:1), the same input resolution settings, the same training epochs and evaluation process, and the same hardware and software operating environment, so as to ensure the repeatability and comparability of the comparison results.

[0209] In step S9, the method for deploying the lightweight model obtained in step S8 to the target terminal device and realizing real-time detection of industrial surface defects is as follows:

[0210] S9A: Exports the lightweight model, trained and subjected to structured pruning and knowledge distillation, into an edge-based inference format. This export includes solidifying the model weights and network structure and generating an inference engine file. In this embodiment, half-precision inference or integer inference can be selected based on the target terminal's computing resources to reduce memory usage and computational overhead. The inference precision includes, but is not limited to, FP16 and INT8, and quantization calibration can be performed using a calibration set to minimize precision loss.

[0211] S9B: Performs consistency checks on the input size, output tensor dimension, and category mapping relationship of the deployed model. If necessary, adjusts the input resolution and preprocessing process according to the actual camera resolution, and synchronously modifies the category configuration file to ensure that the detection category and the display result are consistent. At the same time, configures the post-processing process on the edge. The post-processing includes at least confidence threshold filtering and local density and geometric adaptive nonmaximum suppression to output the final defect category, confidence, and location box information.

[0212] S9C: Integrates the inference engine file and the corresponding category configuration file into the target detection application, completes model file replacement and version management, and generates an installable package after setting the application's name, icon, and permission configuration; the target terminal device includes a mobile terminal or an industrial edge computing terminal, and the installation package can be in APK format or other executable deployment packages.

[0213] S9D: Launch the detection application on the target terminal device, call the camera or image acquisition interface to acquire the industrial surface image stream, perform preprocessing, model inference and postprocessing on each frame of image, output the defect category information, confidence information and location box information in real time, and use the detection results for alarm prompts, quality traceability or production control linkage.

[0214] This specification also provides a space-frequency domain coupled industrial surface defect detection system based on an improved YOLO, including:

[0215] The sample acquisition module is used to acquire images of industrial surface defects.

[0216] The data annotation and segmentation module is used to annotate defect categories and locations and complete dataset segmentation;

[0217] The data augmentation module is used to augment and expand samples and update annotation information synchronously.

[0218] The model structure optimization module is used to introduce spatial wavelet blocks, wavelet-sensory multi-scale spatial modulation, dual-path token fusion module, token wavelet pyramid network and local density and geometric adaptive nonmaximum suppression on the basis of YOLOv5 to build an improved model.

[0219] The model training module is used to train the improved model using the augmented samples;

[0220] The model selection module is used to evaluate and compare multiple candidate models obtained from training and select the optimal model.

[0221] The pruning and distillation module is used to perform structured pruning and knowledge distillation on the optimal model to obtain a lightweight model;

[0222] The deployment and inference module is used to deploy the lightweight model to the target terminal and perform real-time inference to output defect detection results.

[0223] Although the present invention has been described with reference to the above embodiments, those skilled in the art can still make modifications, substitutions, or combinations to the step sequence, module connection method, or parameter settings of the above embodiments without departing from the spirit and principles of the present invention. All modifications that fall within the scope of the claims of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatial-frequency domain coupled industrial surface defect detection method based on improved YOLO, characterized in that, Includes the following steps: Step 1: Collect defect images under the same production conditions and complete the annotation and dataset division; Step 2: Perform data augmentation on the sample images to improve the robustness and generalization ability of the model; Step 3: The YOLOv5 model is structurally innovated by using spatial wavelet blocks, wavelet sensing multi-scale spatial modulation, dual-path token fusion module and token wavelet pyramid network, and local density and geometric adaptive nonmaximum suppression are introduced to optimize the selection of overlapping boxes. Step 4: Perform structured pruning and knowledge distillation on the trained model to obtain a lightweight model, improving deployment efficiency in industrial applications while maintaining detection accuracy; Step 5: Deploy the model in industrial settings to achieve real-time inference and defect location output.

2. The spatial-frequency domain coupled industrial surface defect detection method based on improved YOLO according to claim 1, characterized in that: In step one, the methods for obtaining sample image data include: The LabelImg annotation tool was used to manually annotate the collected industrial surface defect images. For each defect, its category information and spatial location were marked with a minimum bounding rectangle. The boundary of the minimum bounding rectangle was used to completely cover the corresponding defect area to ensure that the defect target was not truncated and the background redundancy was minimized, so as to form annotated sample image data for training and validation.

3. The spatial-frequency domain coupled industrial surface defect detection method based on improved YOLO according to claim 1, characterized in that: In step two, the methods for data augmentation of the sample image data include: Spatial and imaging perturbations are applied to the sample images according to a preset probability to generate augmented samples. The spatial perturbation includes at least one or more of random rotation, scaling, and random cropping, and can be further superimposed with translation or flipping operations. The imaging perturbation includes at least brightness and contrast changes, and can be further superimposed with noise injection or blurring. At the same time, during the process of enhancing and generating new images, the coordinates of the defect annotation box and the defect category are synchronously mapped according to the same transformation rule to maintain consistent annotation, thereby increasing data diversity and improving the model's stable detection capability under different imaging and texture conditions.

4. The spatial-frequency domain coupled industrial surface defect detection method based on improved YOLO according to claim 3, characterized in that: The specific methods in step three include: Spatial wavelet blocks are used to model the input features in parallel with spatial and frequency domain branches and then fuse them. The spatial branch is used to maintain global texture and semantic consistency, while the frequency branch is used to provide high-frequency detail compensation and localization alignment, thereby enhancing the distinguishable representation of minute defects. Wavelet sensing multi-scale spatial modulation is used to jointly adjust the fused features. Low-frequency information guides channel calibration and high-frequency information drives directional gain. Multi-scale convolution is used to achieve synergistic enhancement of channels, multi-scale space and direction, so as to improve the discrimination ability of cross-scale and high aspect ratio defects. The dual-path token fusion module is used to achieve cross-layer feature alignment and selective aggregation, and together with the spatial wavelet block, a token wavelet pyramid network is constructed to progressively fuse multi-layer features under the condition of consistent scale, thereby alleviating the loss of multi-scale defect detail information and enhancing the ability to focus on defect areas. In the prediction output stage, local density and geometric adaptive nonmaximum suppression are introduced to make the suppression intensity of the candidate box adaptively change with the target distribution density and geometry. The area and aspect ratio are compensated and adjusted to reduce over-suppression or under-suppression and stably retain the effective detection results of small defects, extreme aspect ratio defects and multi-scale defects.

5. The spatial-frequency domain coupled industrial surface defect detection method based on improved YOLO according to claim 1, characterized in that: In step four, the methods for obtaining a lightweight model through pruning and distillation include: The improved YOLO model, after training, is subjected to sparsity constraint training to highlight the scalability of redundant structures. The sparsity constraint can be applied to the channel scaling factor and employs... Regular terms, where Defined as: ； in, Scaling factor for each channel, The sparsity strength coefficient is then set, followed by the calculation of the compressibility coefficient. To control the pruning amplitude, the aforementioned Defined as: ； in, The computational load of the model before pruning, To reduce the computational load of the model after pruning, adjust... This approach allows for controllable adjustment of the network slimming ratio. After pruning, the model undergoes structural reconstruction and retraining to restore accuracy. Knowledge distillation is performed using the unpruned model as the teacher model and the post-pruning model as the student model. The distillation loss is defined as: ； in, , The predicted outputs for teachers and students are respectively. The temperature coefficient is the final training objective. ； in, To detect mission losses, The distillation weighting coefficients are used to reduce the number of model parameters and computational load while maintaining defect detection accuracy and improving deployment efficiency as much as possible.

6. The spatial-frequency domain coupling method for detecting industrial surface defects based on improved YOLO according to claim 5, characterized in that: It also includes the following steps: Multiple candidate training models are obtained by performing multiple sets of repeated training or different hyperparameter combinations on sample image data under a preset training strategy. The training strategy includes at least different learning rates, batch sizes, input resolutions, or training rounds. The multiple candidate training models are subjected to unified testing and performance evaluation. Based on the detection accuracy index, inference latency or frame rate index, and spatial-frequency domain coupling feature expression effect index, a comprehensive comparison and screening is performed, and the candidate model with the best comprehensive index is selected as the final deployment training model.

7. The spatial-frequency domain coupling method for detecting industrial surface defects based on improved YOLO according to claim 6, characterized in that: Methods for evaluating and comparing the performance of multiple trained models include: A quantitative test of each candidate model is conducted using an evaluation system for object detection tasks. The evaluation system includes at least precision, recall, mean precision, and mean mean precision to characterize the detection effect. Furthermore, the computational overhead and operational efficiency parameters of the model are statistically analyzed. These parameters include at least floating-point operations, parameter size, inference throughput, and single-frame inference latency. This allows for a comprehensive evaluation and ranking of the model's detection performance and the feasibility of its deployment on the edge.

8. The spatial-frequency domain coupled industrial surface defect detection method based on improved YOLO according to claim 1, characterized in that: In step five, the defect detection methods include: The final training model obtained in step four is converted and integrated into the object detection application. The integration includes replacing the model file in the original detection program with the weight file and network configuration file of the final training model and completing version management. The model's input size, output tensor dimension, and post-processing parameters are checked for consistency. If necessary, the input size, category mapping relationship, and label configuration file are adjusted according to the actual camera resolution and category set to ensure that the detection category is consistent with the display result. The integrated detection program is packaged into an installable application and deployed to the target terminal device, which includes a mobile terminal or an industrial edge computing terminal. The application is launched to call the final trained model to perform online inference and output defect category, confidence level and location box information to achieve real-time detection of industrial surface defects.

9. A spatial-frequency domain coupled industrial surface defect detection system based on improved YOLO, used to execute the spatial-frequency domain coupled industrial surface defect detection method based on improved YOLO as described in any one of claims 1-8, characterized in that, include: The sample acquisition module is used to acquire images of industrial surface defects. The data annotation and segmentation module is used to annotate defect categories and locations and complete dataset segmentation; The data augmentation module is used to augment and expand samples and update annotation information synchronously. The model structure optimization module is used to introduce spatial wavelet blocks, wavelet-sensory multi-scale spatial modulation, dual-path token fusion module, token wavelet pyramid network and local density and geometric adaptive nonmaximum suppression on the basis of YOLOv5 to build an improved model. The model training module is used to train the improved model using the augmented samples; The model selection module is used to evaluate and compare multiple candidate models obtained from training and select the optimal model. The pruning and distillation module is used to perform structured pruning and knowledge distillation on the optimal model to obtain a lightweight model; The deployment and inference module is used to deploy the lightweight model to the target terminal and perform real-time inference to output defect detection results.

Images