Metal surface micro-defect detection method and system based on deep learning

Abstract

This application relates to a method and system for metal surface micro-defect detection based on deep learning. The method comprises: obtaining a surface image of the metal to be inspected and performing preprocessing; calling a trained defect detection model to perform defect detection on the surface image to obtain a bounding box of the defect area in the surface image; wherein the trained defect detection model includes an attention combination module, an enhanced feature extraction module, and a four-branch stacking module; the attention combination module is used to extract multi-scale features of the surface image; the enhanced feature extraction module extracts deep local features of the surface image by expanding the receptive field; the four-branch stacking module reduces the number of model parameters through network structure optimization; and determining the actual defect size of the metal to be inspected based on the size information of the bounding box. This method can improve the efficiency of metal surface micro-defect detection.

Description

Technical Field

[0001] The present application relates to the field of non-destructive testing technology, and in particular to a method and system for detecting metal surface micro-defects based on deep learning. Background Art

[0002] With the acceleration of industrialization, metal materials play an indispensable role in numerous industries due to their excellent mechanical properties and cost-effectiveness. However, during the production process, defects such as cracks, scratches, and inclusions often inevitably appear on the metal surface. These defects can reduce the material's toughness and strength, thereby affecting the safety and service life of metal products. Therefore, the key to improving product quality is to non-destructively and efficiently detect micro-defects on metal surfaces during the production process.

[0003] In related technologies, common metal surface micro-defect detection methods include metallography and scanning electron microscopy. These two methods obtain material structure and defect information by observing detection photos, but there are problems such as difficulty in obtaining detection photos and complex operations, which affect detection efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide a metal surface micro-defect detection method and system based on deep learning to address the technical problem of low detection efficiency of the above-mentioned method.

[0005] In a first aspect, the present application provides a method for detecting metal surface micro-defects based on deep learning. The method comprises:

[0006] Acquire the surface image of the metal to be detected and perform preprocessing;

[0007] Calling a trained defect detection model to perform defect detection on the surface image to obtain a bounding box of a defect area in the surface image;

[0008] Determining the actual defect size of the metal to be inspected based on the size information of the bounding box;

[0009] Among them, the trained defect detection model includes an attention combination module, an enhanced feature extraction module, and a four-branch stacking module; the attention combination module is used to extract the multi-scale features of the surface image; the enhanced feature extraction module extracts the deep local features of the surface image by expanding the receptive field; the four-branch stacking module reduces the number of model parameters through network structure optimization.

[0010] In one embodiment, the acquisition of the surface image of the metal to be detected and the preprocessing are specifically as follows:

[0011] Use an ultrasonic scanning microscope equipped with a 300MHz high-frequency probe and pure distilled water as a coupling agent to perform C-Scan scanning with a step size of 1μm to obtain high-resolution metal surface images;

[0012] The collected high-resolution images are cropped into images of the same size, and the cropped image data is cleaned, including removing images that are unqualified after cropping and images containing noise introduced during the scanning process.

[0013] In one embodiment, the trained defect detection model includes a backbone network, a neck network, and a detection head connected in sequence;

[0014] The attention combination module is provided in the backbone network and the neck network;

[0015] The enhanced feature extraction module is provided in the neck network;

[0016] The four-branch stacking module is arranged in the neck network.

[0017] The attention combination module includes a multi-scale attention submodule and a downsampling feature extraction submodule;

[0018] The multi-scale attention submodule is used to extract features by reshaping and grouping channels, encoding global information and recalibrating channel weights through parallel branches, and aggregating output features of parallel branches across dimensions.

[0019] In one embodiment, the enhanced feature extraction module is obtained by adjusting the spatial pyramid pooling cross-stage partial connection module through the spatial pyramid fast pooling principle; and is used to extract deep local features through multi-scale pooling and cross-stage connection.

[0020] In one embodiment, the enhanced feature extraction module includes two branches. In the first branch, the input content is processed by multiple convolution blocks to obtain a first output. The first output is subjected to three consecutive pooling operations. The first output is concatenated with the output results obtained by each pooling operation and then processed by two convolution blocks to obtain the first branch result.

[0021] In the second branch, the input content is processed by a convolution block to obtain a second branch result;

[0022] After the first branch result and the second branch result are spliced together, they are processed by a convolution block to obtain the output result of the enhanced feature extraction module.

[0023] In one embodiment, the four-branch stacking module optimizes the network structure by adjusting the number and distribution of 1×1 convolutions and 3×3 convolutions, and reducing splicing branches.

[0024] In one embodiment, the four-branch stacking module includes two branches;

[0025] In the first branch, the input content is processed by a 1×1 convolution block to obtain the first output;

[0026] In the second branch, the input content is processed by a 1×1 convolution block to obtain a second output, the second output is processed by a 3×3 convolution block to obtain a third output, the second output and the third output are concatenated and input into a 1×1 convolution block to obtain a fourth output; the fourth output is processed by a 3×3 convolution block to obtain a fifth output, and the fifth output is further processed by a 3×3 convolution block to obtain a sixth output;

[0027] The first output, the fourth output, the fifth output and the sixth output are spliced and input into a 1×1 convolution block for processing to obtain the output result of the four-branch stacking module.

[0028] In one embodiment, determining the actual defect size of the metal to be inspected based on the size information of the bounding box includes:

[0029] Obtaining a predetermined size equation; the size equation is obtained by fitting the actual defect size of the sample surface image with the size information of the boundary box of the defect area in the sample surface image;

[0030] Substituting the size information of the bounding box into the size equation, the actual defect size of the metal to be inspected is obtained.

[0031] In one embodiment, the defect detection model is trained in the following manner:

[0032] Acquire a sample data set; the sample data set includes a sample surface image and a true bounding box of the sample surface image;

[0033] Inputting the sample surface image into an initial defect detection model to perform defect detection and obtain a predicted bounding box;

[0034] Obtaining an intersection-over-union (IoU) ratio between the predicted bounding box and the true bounding box, and a shape similarity between the predicted bounding box and the true bounding box, and determining a loss value between the predicted bounding box and the true bounding box based on the IoU ratio and the shape similarity;

[0035] The initial defect detection model is trained with reducing the loss value as a training goal to obtain the trained defect detection model.

[0036] In a second aspect, the present application also provides a metal surface micro-defect detection system based on deep learning. The system comprises:

[0037] An acquisition unit, which scans the metal surface to be detected with an ultrasonic scanning microscope, obtains a surface image of the metal to be detected and performs preprocessing;

[0038] The detection unit calls a defect detection model trained on a computer device to perform defect detection on the surface image to obtain a bounding box of a defect area in the surface image; wherein the trained defect detection model includes an attention combination module, an enhanced feature extraction module, and a four-branch stacking module; the attention combination module is used to extract multi-scale features of the surface image; the enhanced feature extraction module extracts deep local features of the surface image by expanding the receptive field; and the four-branch stacking module reduces the number of model parameters by optimizing the network structure;

[0039] A determining unit determines an actual defect size of the metal to be detected based on the size information of the bounding box.

[0040] The above-mentioned deep learning-based metal surface micro-defect detection method and system obtains and preprocesses the surface image of the metal to be inspected, then calls a trained defect detection model to perform defect detection on the surface image, obtaining a bounding box of the defect area in the surface image; based on the size information of the bounding box, the actual defect size of the metal to be inspected is determined. Through automated image processing and model reasoning, metal surface defects can be quickly determined, improving detection efficiency. Furthermore, an attention combination module, an enhanced feature extraction module, and a four-branch stacking module are constructed in the defect detection model; the attention combination module can improve the detection capability of defects of different sizes by performing multi-scale feature extraction; the enhanced feature extraction module can improve the characterization capability of complex defects by expanding the receptive field and extracting deep local features; and the four-branch stacking module can reduce the number of model parameters through network structure optimization, making the model more suitable for deployment on resource-constrained devices. Therefore, through the collaborative work of the attention combination module, the enhanced feature extraction module, and the four-branch stacking module, efficient, accurate and lightweight defect detection can be achieved, and then the trained defect detection model can have a good detection effect on micro-defects on the metal surface, and can achieve high-precision recognition of tiny defects of 10um and above on the metal surface, effectively improving the accuracy and reliability of micro-defect detection on the metal surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a flow chart of a method for detecting metal surface micro-defects based on deep learning in one embodiment;

[0042] Figure 2 A schematic diagram of the structure of an ultrasonic scanning system in one embodiment;

[0043] Figure 3 Schematic diagram of the structure of an improved YOLOv7 network in one embodiment;

[0044] Figure 4 Schematic diagram of the structure of an efficient multi-scale attention submodule in one embodiment;

[0045] Figure 5 is a structural diagram of an improved enhanced feature extraction module in one embodiment;

[0046] Figure 6 This is a schematic structural diagram of a four-branch stacking module in one embodiment;

[0047] Figure 7 A scatter plot of the predicted box diagonal length and the actual longest side length of the inclusion in one embodiment;

[0048] Figure 8 1 is a flow chart of a defect detection model training method according to an embodiment;

[0049] Figure 9 A comparison diagram of different processing results in one embodiment;

[0050] Figure 10 A comparison chart of the actual detection effects before and after the improvement of the YOLOv7 network in one embodiment;

[0051] Figure 11 1 is a structural block diagram of a metal surface micro-defect detection system based on deep learning in one embodiment;

[0052] Figure 12 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] In one embodiment, Figure 1 As shown, a method for detecting micro-defects on metal surfaces based on deep learning is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers. The server can be implemented as an independent server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0055] Step S110 , obtaining a surface image of the metal to be detected and performing preprocessing.

[0056] In practice, ultrasonic microscopy can be used to capture images of metal surfaces. Ultrasonic microscopy, a type of ultrasonic testing technology, can capture high-resolution images of tiny defects on metal surfaces through scanning imaging, meeting the basic need for detecting tiny defects through imaging.

[0057] Specifically, an ultrasonic scanning microscope can be used to capture the surface image of the metal to be inspected. The ultrasonic scanning microscope can be equipped with a high-frequency probe (such as 300MHz) and use pure distilled water as a coupling agent to perform C-Scan scanning with a preset step size (such as 1um) to obtain high-resolution metal surface images. Figure 2 The ultrasonic scanning system structure diagram shown in the figure is realized by the propagation and reflection of high-frequency ultrasonic waves inside the metal to be detected. The ultrasonic waves are emitted by the transducer, focused by the acoustic lens and transmitted to the metal to be detected. When the ultrasonic waves encounter the interface of different media, due to the difference in acoustic impedance, reflected waves are generated. These reflected waves are received by the transducer and converted into electrical signals. After a certain amount of processing, the electrical signals are transmitted digitally to the computer for signal and image processing.

[0058] When ultrasonic waves propagate in a material, their wave equation can be expressed as:

[0059] (1)

[0060] Where: is the amplitude of the ultrasonic wave; is length and satisfies .

[0061] The reflected wave generated by ultrasonic wave on the transmitting plane is expressed as:

[0062] (2)

[0063] If there are no defects inside the metal to be tested, the ultrasonic wave can continue to propagate in the metal until it encounters the bottom surface of the metal to be tested and is emitted. At this time, the bottom surface reflection wave is expressed as:

[0064] (3)

[0065] When there is a defect in the metal to be detected and there is a difference in acoustic impedance between the defect and the metal to be detected, the ultrasonic wave will A reflection is generated, and the reflected signal is: (4)

[0066] From the above formula, we can know that the reflected wave of ultrasonic echo at the defect lags behind the reflected wave of the transmitting plane. , ahead of the reflected wave from the lower surface By using the phase and amplitude information of each echo point to display it on the screen in color or grayscale pixels, an ultrasonic scanning image can be obtained.

[0067] In a specific implementation, the collected metal surface images can be preprocessed by cropping the images into pictures of the same size and cleaning the cropped picture data, including removing unqualified pictures after cropping and pictures containing noise introduced during the scanning process.

[0068] Step S120 , calling the trained defect detection model to perform defect detection on the surface image, and obtaining a boundary box of the defect area in the surface image.

[0069] The defect detection model can be an improved model based on YOLOv7. Specifically, the improvements to YOLOv7 include introducing an efficient multi-scale attention mechanism (EMA) and combining it with a downsampling feature extraction submodule to form an attention combination module, improving the SPPCSPC module in the neck network to form an enhanced feature extraction module, and designing a four-branch stacking module to replace all ELAN-H modules in the neck network.

[0070] Among them, the attention combination module is used to extract multi-scale features of surface images; the enhanced feature extraction module extracts deep local features of surface images by expanding the receptive field; and the four-branch stacking module reduces the number of model parameters by optimizing the network structure.

[0071] Among them, the bounding box includes the position of the bounding box (such as the xy coordinates of the upper left corner) and the size of the bounding box (length and width), for example, expressed as bbox=(x,y,w,h).

[0072] In a specific implementation, a modified YOLOv7 model can be pre-built as the initial defect detection model. Surface images of sample metals are collected and the ground-truth bounding boxes of their defective areas are annotated to construct a sample dataset. This sample dataset is used to train and test the initial defect detection model, resulting in a trained defect detection model. After training, the trained defect detection model can be invoked to perform defect detection on surface images of the metal to be inspected, outputting the bounding boxes of the detected defective areas.

[0073] Step S130 : determining the actual defect size of the metal to be inspected based on the size information of the bounding box.

[0074] In a specific implementation, the bounding box information for the defect area in the surface image output by the trained defect detection model includes the length and width of the bounding box. The diagonal length of the bounding box is calculated from this length and width. Based on this diagonal length, the length of the longest side of the actual defect in the metal to be inspected is determined. The defect size of the metal to be inspected can be graded based on the length of the longest side of the actual defect. For example, based on the actual defect size of the metal to be inspected, multiple preset size ranges can be queried to determine a target size range that matches the actual defect size. Each size range has a corresponding level, and the level corresponding to the target size range can be used as the defect level corresponding to the metal to be inspected.

[0075] In the above-mentioned deep learning-based metal surface micro-defect detection method, a surface image of the metal to be inspected is obtained and preprocessed, and then a trained defect detection model is called to perform defect detection on the surface image to obtain a bounding box of the defect area in the surface image. Based on the size information of the bounding box, the actual defect size of the metal to be inspected is determined. Through automated image processing and model inference, metal surface defects can be quickly determined, improving detection efficiency. Furthermore, an attention combination module, an enhanced feature extraction module, and a four-branch stacking module are constructed in the defect detection model. The attention combination module can improve the detection capability of defects of different sizes by performing multi-scale feature extraction. The enhanced feature extraction module can improve the characterization capability of complex defects by expanding the receptive field and extracting deep local features. The four-branch stacking module can reduce the number of model parameters through network structure optimization, making the model more suitable for deployment on resource-constrained devices. Therefore, through the collaborative work of the attention combination module, the enhanced feature extraction module, and the four-branch stacking module, efficient, accurate and lightweight defect detection can be achieved, and then the trained defect detection model can have a good detection effect on micro-defects on the metal surface, and can achieve high-precision recognition of tiny defects of 10um and above on the metal surface, effectively improving the accuracy and reliability of micro-defect detection on the metal surface.

[0076] In an exemplary embodiment, the trained defect detection model includes a backbone network, a neck network and a detection head connected in sequence; wherein, the attention combination module is arranged in the backbone network and the neck network; the enhanced feature extraction module is arranged in the neck network; and the four-branch stacking module is arranged in the neck network.

[0077] refer to Figure 3 , is a schematic diagram of the structure of the defect detection model provided in this application, which is obtained by improving the YOLOv7 network. Figure 3 As shown in Figure 1, the defect detection model includes a backbone network, a neck network, and a detection head. The three networks are connected in sequence. The output of the backbone network is the input of the neck network, and the output of the neck network is the input of the detection head. Figure 3The attention combination module in the model is used to extract multi-scale features; the enhanced feature extraction module is used to expand the receptive field and extract deep local features of the surface image; and the four-branch stacking module is used to reduce the number of model parameters through network structure optimization. There are multiple attention combination modules and four-branch stacking modules, and one enhanced feature extraction module. The attention combination module is set in the backbone network and the neck network, while the enhanced feature extraction module is set in the neck network and the four-branch stacking module is also set in the neck network.

[0078] in addition, Figure 3 The CBS module in YOLOv7 consists of Conv (convolution) + BatchNorm (batch normalization) + SiLU (activation function), which is used to extract local features and perform nonlinear transformations. The Efficient Feature Aggregation Module (ELAN) is an efficient feature aggregation module introduced in YOLOv7 to enhance the multi-scale representation of features. The ELAN module fuses features at different levels through a multi-branch structure to improve model performance. In the figure, splicing is an operation used for feature fusion, which splices multiple feature maps in the channel dimension. Upsampling is an operation used to increase the spatial resolution of feature maps, usually achieved through interpolation methods. Reparameterized convolution is a convolution optimization technique that combines multiple convolution operations into a single convolution operation through reparameterization, thereby improving inference speed.

[0079] In this embodiment, the attention combination module improves the detection capability of defects of different sizes by performing multi-scale feature extraction. The enhanced feature extraction module improves the characterization capability of complex defects by expanding the receptive field and extracting deep local features. The four-branch stacking module optimizes the network structure to reduce the number of model parameters and computational cost, making the model more suitable for deployment on resource-constrained devices. Thus, the collaborative work of the attention combination module, the enhanced feature extraction module, and the four-branch stacking module enables efficient, accurate, and lightweight defect detection.

[0080] In an exemplary embodiment, the attention combination module includes a multi-scale attention submodule and a downsampling feature extraction submodule. The multi-scale attention submodule is configured to extract features by reshaping and grouping channels, encoding global information and recalibrating channel weights through parallel branches, and aggregating the output features of parallel branches across dimensions.

[0081] Specifically, the attention combination module consists of an efficient multi-scale attention submodule (EMA) and a downsampling feature extraction submodule. EMA is a network architecture designed to improve the feature extraction capabilities of convolutional neural networks. This architecture achieves multi-scale feature optimization through three key steps: first, channel reorganization techniques are used to group feature maps to form multi-granular feature representations; then, parallel spatial and channel attention branches are used to capture global contextual information and inter-channel dependencies, respectively; finally, a cross-dimensional interaction mechanism is introduced to dynamically fuse feature responses from different branches. Compared to traditional attention mechanisms, EMA offers significant computational efficiency advantages and enables more accurate feature calibration.

[0082] refer to Figure 4 , a schematic diagram of the multi-scale attention submodule in one embodiment. This submodule reshapes some channels into batch dimensions, grouping the channel dimensions into multiple sub-features to ensure a good distribution of spatial semantic features within each feature group. Furthermore, in addition to encoding global information in parallel branches to recalibrate channel weights, it also aggregates the output features of two parallel branches through cross-dimensional interactions to capture pixel-level pairwise relationships, thereby improving feature extraction while avoiding the side effects of channel dimension reduction.

[0083] In this embodiment, an efficient multi-scale attention mechanism is introduced to construct an EMA submodule, which is combined with the downsampling feature extraction submodule to form an attention combination module. This can effectively capture the multi-scale features of the image without reducing the channel dimension, thereby improving the feature extraction effect.

[0084] In an exemplary embodiment, the enhanced feature extraction module is obtained by adjusting the spatial pyramid pooling cross-stage partial connection module through the spatial pyramid fast pooling principle, and is used to extract deep local features through multi-scale pooling and cross-stage connection.

[0085] Specifically, the enhanced feature extraction module draws on the Spatial Pyramid Pooling Fast (SPPF) principle to improve the Spatial Pyramid Pooling Cross Stage Partial Connections (SPPCSPC module).

[0086] More specifically, in an exemplary embodiment, the enhanced feature extraction module includes two branches; in the first branch, the input content is processed by multiple convolution blocks to obtain a first output, the first output is subjected to three consecutive pooling operations, the first output is spliced with the output result obtained by each pooling operation, and then processed by two convolution blocks to obtain the first branch result; in the second branch, the input content is processed by a convolution block to obtain a second branch result; the first branch result and the second branch result are spliced and then processed by a convolution block to obtain the output result of the enhanced feature extraction module.

[0087] refer to Figure 5 , is a structural diagram of an enhanced feature extraction module shown in one embodiment. The enhanced feature extraction module includes two branches. The first branch obtains a first output after being processed by three CBS convolution blocks (the convolution kernels are 1×1, 3×3, and 1×1, respectively). The first output is sequentially subjected to three maximum pooling operations with a pooling kernel size of 5, and a result is output after each pooling. The first output is concatenated with the results of the three pooling outputs, and then processed by two CBS convolution blocks (the convolution kernels are 1×1 and 3×3, respectively) to obtain the first branch result. The second branch is processed by a 1×1 CBS convolution block to obtain the second branch result. After the first branch result and the second branch result are concatenated and processed by a 1×1 CBS convolution block, the output result of the enhanced feature extraction module is obtained. The original SPPCSPC module performs three maximum pooling operations on the first output with pooling kernel sizes of 5, 9, and 13, and then concatenates the output results after each pooling with the first output. Compared with it, the enhanced feature extraction module obtains more deep features by adjusting the pooling kernel size and deeper pooling, thereby improving the model training effect.

[0088] Among them, the CBS module is composed of a convolutional layer (Conv), a batch normalization layer (BatchNorm) and a SiLU (activation function) in sequence, which is used to extract local features and perform nonlinear transformations.

[0089] In this embodiment, the SPPF design idea is used to improve the SPPCSPC module in the neck network to form an enhanced feature extraction module. By expanding the receptive field, more deep local features can be extracted, thereby achieving more accurate recognition of surface images and improving the confidence of subsequent surface image defect detection.

[0090] In an exemplary embodiment, the four-branch stacking module optimizes the network structure by adjusting the number and distribution of 1×1 convolutions and 3×3 convolutions, and reducing splicing branches.

[0091] Specifically, the four-branch stacking module includes two branches; in the first branch, the input content is processed by a 1×1 convolution block to obtain a first output; in the second branch, the input content is processed by a 1×1 convolution block to obtain a second output, and the second output is processed by a 3×3 convolution block to obtain a third output, and the second output and the third output are spliced and input into a 1×1 convolution block to obtain a fourth output; the fourth output is processed by a 3×3 convolution block to obtain a fifth output, and the fifth output is further processed by a 3×3 convolution block to obtain a sixth output; the first output, the fourth output, the fifth output and the sixth output are spliced and input into a 1×1 convolution block for processing to obtain the output result of the four-branch stacking module.

[0092] refer to Figure 6 , a schematic diagram of the structure of a four-branch stacked module in one embodiment. The CBS module in the figure consists of Conv (convolution) + BatchNorm (batch normalization) + SiLU (activation function), which is used to extract local features and perform nonlinear transformations. The four-branch stacked module specifically includes two branches. The first branch passes through a 1×1 CBS convolution block to obtain the first output; the second branch passes through a 1×1 CBS convolution block to obtain the second output, and the second output passes through a 3×3 CBS convolution block to obtain the third output. The second and third outputs are concatenated and passed through a 1×1 CBS convolution block to obtain the fourth output. The fourth output then passes through a 3×3 CBS convolution block to obtain the fifth output, and the fifth output passes through another 3×3 CBS convolution block to obtain the sixth output. Finally, the first, fourth, fifth, and sixth outputs are concatenated and passed through a 1×1 CBS convolution block to obtain the final output. Compared to the traditional ELAN-H module, the four-branch stacked module contains three 3×3 convolutions and three 1×1 convolutions, while the ELAN-H module has four 3×3 convolutions and two 1×1 convolutions. The 3×3 and 1×1 refer to the convolution kernel sizes, which are essentially parameters that require training. Furthermore, the four-branch stacked module has four branches spliced together at the end, while the ELAN-H module has six branches spliced together, reducing the number of branches by two. Therefore, the four-branch stacked module reduces the number of model parameters by adjusting the number and distribution of 1×1 and 3×3 convolutions and reducing the number of spliced branches at the end.

[0093] In this embodiment, by designing a four-branch stacked module to replace all ELAN-H modules in the neck network, the number of model parameters and computational cost can be reduced while maintaining the training effect unchanged.

[0094] In an exemplary embodiment, the above-mentioned step S130 determines the actual defect size of the metal to be inspected based on the size information of the bounding box, including: obtaining a predetermined size equation; the size equation is obtained by fitting the actual defect size of the sample surface image with the size information of the bounding box of the defect area in the sample surface image; and substituting the size information of the bounding box into the size equation to obtain the actual defect size of the metal to be inspected.

[0095] Specifically, considering the irregular shapes of metal surface defects and the varying effects of defects of different sizes on metal material properties, the more practical defect classification method uses the longest side length of the defect. The specific steps are as follows: a scatter plot is drawn based on the diagonal length of the predicted bounding box of several sample points and the actual longest side length of the defect to confirm their linear correlation. The least squares method is then used to obtain a linear fit equation, which serves as the size equation. The size information of the bounding box obtained from subsequent actual inspections is then substituted into this size equation to determine the actual defect size of the metal being inspected.

[0096] The core idea of least squares linear fitting is to find a straight line that minimizes the sum of the squares of the vertical distances (i.e., errors) from all data points to the straight line. Assume that the fitting straight line between the diagonal length of the predicted box and the longest side length of the actual inclusion is , then for each , its fitting value is , so the sum of squared errors is:

[0097] (5)

[0098] In order to seek When the minimum value is obtained and , respectively for and Partial derivative of:

[0099] (6)

[0100] (7)

[0101] Let the two partial derivatives be zero and solve to get the equation of the fitted line. Substitute the data and solve to get the fitted equation: , is the actual longest side length of the defect, is the diagonal length of the prediction box, which is drawn in the scatter plot, such as Figure 7 As shown. The coefficient of determination of the fitting equation The goodness of fit of the equation is 0.89. Ten sample points were selected and the defect size was calculated using the fitting equation. The specific data is shown in the table below. The fitting results show that the fitting equation can effectively classify defect sizes.

[0102] Table 1 Fitting data of selected sample points

[0103]

[0104] In this embodiment, taking into account the irregularity of defect shapes and the differences in the effects of defects of different sizes on metal properties, the process of determining the actual defect size of the metal to be detected based on the length of the longest side of the defect as the bounding box size information can provide a more accurate evaluation method for defect detection of metal materials.

[0105] In an exemplary embodiment, Figure 8 As shown in Figure 2, the defect detection model is trained in the following way:

[0106] Step S810, obtaining a sample data set; the sample data set includes a sample surface image and a true bounding box of the sample surface image;

[0107] Step S820 , inputting the sample surface image into the initial defect detection model to perform defect detection and obtain a predicted bounding box;

[0108] Step S830: Obtain the intersection-over-union (IoU) between the predicted bounding box and the true bounding box, as well as the shape similarity between the predicted bounding box and the true bounding box, and determine the loss value between the predicted bounding box and the true bounding box based on the IoU and the shape similarity.

[0109] In step S840 , the initial defect detection model is trained with reducing the loss value as the training goal to obtain a trained defect detection model.

[0110] In a specific implementation, batches of sample surface images can be collected in advance. These images can be obtained by scanning the sample metal using an ultrasonic scanning microscope or by using historically scanned metal surface images. Each sample surface image is annotated, for example, using annotation software. The sample surface images and their annotation information form a sample dataset. The sample dataset is then randomly divided into training, test, and validation sets according to a preset ratio (e.g., 8:1:1) for model training, testing, and validation, respectively.

[0111] During each training process, the sample surface image is input into the initial defect detection model for defect detection, and the predicted bounding box obtained by detecting the defect area is output. The loss value is calculated based on the predicted bounding box and the annotated true bounding box. Specifically, the intersection-over-union ratio between the predicted bounding box and the true bounding box, as well as the shape similarity between the predicted bounding box and the true bounding box are obtained. The intersection-over-union ratio and shape similarity are substituted into the loss function to calculate the loss value between the predicted bounding box and the true bounding box. If the loss value is greater than the threshold, the model parameters are adjusted with the loss value as the training goal to obtain a new defect detection model. The next sample surface image is input into the new defect detection model again for detection, the loss value is calculated, and the step of comparing with the threshold is returned until the loss value converges or reaches the preset number of training times. The training ends and a trained defect detection model is obtained.

[0112] Among them, the calculation formula of the loss function can be expressed as:

[0113] (8)

[0114] Among them, IoU represents the intersection-over-union ratio between the predicted bounding box and the true bounding box. 、 Represents the shape similarity between the predicted bounding box and the true bounding box.

[0115] The calculation formula of intersection-union ratio can be expressed as: (9)

[0116] Among them, A is the real bounding box and B is the predicted bounding box.

[0117] The calculation process is as follows:

[0118] (10)

[0119] (11)

[0120] (12)

[0121] in, , represents the center coordinates of the predicted bounding box, , represents the center coordinates of the true bounding box, Represents the diagonal length of the minimum bounding rectangle that can contain both the predicted bounding box and the true bounding box. , represent the width and height of the ground-truth bounding box, is a scaling factor related to the size of the target defect in the dataset.

[0122] The calculation formula is as follows:

[0123] (13)

[0124] (14)

[0125] in, , Represents the width and height of the predicted bounding box.

[0126] In some embodiments, after obtaining the sample data set, the sample data set can be preprocessed first. For example, each sample surface image in the sample data set is cropped into images of the same size, and the cropped images are cleaned. The cleaning includes removing unqualified images after cropping and images containing noise introduced during the scanning process, thereby obtaining a preprocessed sample data set, and performing model training using the preprocessed sample data set.

[0127] In this embodiment, during the defect detection model training process, when calculating the loss between the predicted bounding box and the true bounding box, not only the overlapping area of the bounding box is considered, but also a shape similarity measurement is introduced, which can more accurately measure the degree of matching between the predicted bounding box and the true bounding box, and make the regression of the bounding box more accurate, so as to better capture the detailed features of small targets and improve the detection accuracy of small targets, thereby realizing the detection of micro-defects in the metal surface image to be inspected.

[0128] In one embodiment, when training and testing a defect detection model, a Pytorch deep learning framework can be used to train and test the model; for example, cuda11.3 + torch1.12.1 + torchvision0.13.1 is selected as the deep learning framework, and a Python3.9.16 compilation environment is deployed; the input surface image size is 992×992, the maximum learning rate during training is set to 0.001, the minimum learning rate is set to 0.00001, and the cosine annealing learning rate decay method is used to gradually reduce the learning rate from 0.001 to 0.00001 throughout the training process; the batch size is set to 16, the Adam (Adaptive Moment Estimation) optimizer is used, the momentum is set to 0.9, and the number of training rounds is set to 300 rounds; during the training process, a mosaic (creating a new composite image by stitching multiple images together) data augmentation method is combined with a mixup (linearly combining two images) data augmentation method to improve the robustness of the model.

[0129] To assess the accuracy and stability of the model, we used mean average precision (mAP) and frames per second (FPS) as evaluation metrics. mAP is a comprehensive evaluation metric that combines the precision and recall of different categories to measure the model's detection accuracy. A higher mAP value indicates higher prediction accuracy. A threshold of 0.5 was used for the intersection-over-union (IoU) ratio. FPS represents the speed at which the model processes frames per second and is used to measure the model's recognition speed. A higher mAP value indicates faster prediction speed.

[0130] Table 2 Comparative experimental results

[0131]

[0132] As can be seen from Table 2, compared with the traditional YOLOv7 model, the improved YOLOv7 model in this application has improved the detection accuracy from 95.3% to 98.2% on the basis of reducing the number of network parameters and the amount of calculation, and the recognition speed has also increased by 5%, which verifies the effectiveness of the improved model in this application.

[0133] Figure 9 The following chart compares different processing results: (a) shows the manual labeling result, (b) the result processed by the ultrasonic scanning microscope's built-in evaluation system, (c) the prediction result using the original YOLOv7 model, and (d) the prediction result using the improved YOLOv7 model in this application. (b) The ultrasonic scanning microscope's built-in evaluation system uses a relatively simple threshold segmentation method for defect analysis, which is affected by the brightness of the scanned image and has poor applicability, making it unable to accurately analyze defects. (d) Compared with the traditional YOLOv7 model prediction result in (c), the improved YOLOv7 model prediction result in (d) not only accurately identifies all inclusions, but also improves the recognition confidence.

[0134] Figure 10 This is a comparison chart of the detection quality of the traditional YOLOv7 model and the improved YOLOv7 model. 50 new pre-processed images were selected and numbered, and then input into the improved YOLOv7 model and the traditional YOLOv7 model for prediction. The results were compared with the manual annotation results, and the number of false detections and missed detections for every 10 images were counted. Figure 10 The results are shown in the figure. It can be seen from the figure that the improved YOLOv7 model actually performs better than the original YOLOv7 model. Overall, the improved YOLOv7 model has better detection performance.

[0135] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0136] Based on the same inventive concept, the embodiments of the present application also provide a metal surface micro-defect detection system for implementing the above-mentioned metal surface micro-defect detection method based on deep learning. The implementation solution provided by this system is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations of one or more embodiments of the metal surface micro-defect detection system based on deep learning provided below can be referred to the limitations of the metal surface micro-defect detection method based on deep learning above, and will not be repeated here.

[0137] In one embodiment, Figure 11 As shown, a metal surface micro-defect detection system based on deep learning is provided, including:

[0138] An acquisition unit 1101 scans the metal surface to be inspected with an ultrasonic scanning microscope to acquire a surface image of the metal to be inspected and performs preprocessing;

[0139] Detection unit 1102 calls a defect detection model trained on a computer device to perform defect detection on the surface image to obtain a bounding box of the defect area in the surface image; wherein the trained defect detection model includes an attention combination module, an enhanced feature extraction module, and a four-branch stacking module; the attention combination module is used to extract multi-scale features of the surface image; the enhanced feature extraction module extracts deep local features of the surface image by expanding the receptive field; and the four-branch stacking module reduces the number of model parameters by optimizing the network structure;

[0140] The determining unit 1103 determines the actual defect size of the metal to be detected based on the size information of the bounding box.

[0141] In one embodiment, the trained defect detection model includes a backbone network, a neck network and a detection head connected in sequence; the attention combination module is arranged in the backbone network and the neck network; the enhanced feature extraction module is arranged in the neck network; and the four-branch stacking module is arranged in the neck network.

[0142] In one embodiment, the attention combination module includes a multi-scale attention submodule and a downsampling feature extraction submodule. The multi-scale attention submodule adopts a three-stage feature optimization mechanism to achieve efficient feature enhancement. First, a multi-granularity feature expression is established through intelligent reshaping and grouping processing of the channel dimension; secondly, a parallel branch architecture is used to synchronously capture global context information, and adaptive reweighting of feature channels is achieved based on the channel attention mechanism; finally, through an innovative cross-dimensional feature aggregation strategy, the multi-scale features output by each branch are deeply fused to construct an enhanced feature representation with rich semantic information.

[0143] In one embodiment, the enhanced feature extraction module is obtained by adjusting the spatial pyramid pooling cross-stage partial connection module through the spatial pyramid fast pooling principle; and is used to extract deep local features through multi-scale pooling and cross-stage connection.

[0144] In one embodiment, the enhanced feature extraction module includes two branches; in the first branch, the input content is processed by multiple convolution blocks to obtain a first output, the first output is subjected to three consecutive pooling operations, the first output is spliced with the output result obtained by each pooling operation, and then processed by two convolution blocks to obtain the first branch result; in the second branch, the input content is processed by a convolution block to obtain a second branch result; the first branch result and the second branch result are spliced and then processed by a convolution block to obtain the output result of the enhanced feature extraction module.

[0145] In one embodiment, the four-branch stacking module optimizes the network structure by adjusting the number and distribution of 1×1 convolutions and 3×3 convolutions, and reducing splicing branches.

[0146] In one embodiment, the four-branch stacking module includes two branches; in the first branch, the input content is processed by a 1×1 convolution block to obtain a first output; in the second branch, the input content is processed by a 1×1 convolution block to obtain a second output, the second output is processed by a 3×3 convolution block to obtain a third output, the second output and the third output are spliced and input into a 1×1 convolution block to obtain a fourth output; the fourth output is processed by a 3×3 convolution block to obtain a fifth output, and the fifth output is further processed by a 3×3 convolution block to obtain a sixth output; the first output, the fourth output, the fifth output and the sixth output are spliced and input into a 1×1 convolution block to obtain the output result of the four-branch stacking module.

[0147] In one embodiment, the computer device may be a server, and its internal structure diagram may be as follows: Figure 12 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the process of metal surface micro-defect detection based on deep learning. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a metal surface micro-defect detection method based on deep learning is implemented.

[0148] Those skilled in the art will understand that Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0149] In one embodiment, the determination unit 1103 is further used to obtain a predetermined size equation; the size equation is obtained by fitting the actual defect size of the sample surface image with the size information of the bounding box of the defect area in the sample surface image; the size information of the bounding box is substituted into the size equation to obtain the actual defect size of the metal to be detected.

[0150] In one embodiment, the system also includes a model training module for obtaining a sample data set; the sample data set includes a sample surface image and a real bounding box of the sample surface image; the sample surface image is input into the initial defect detection model for defect detection to obtain a predicted bounding box; the intersection-over-union ratio between the predicted bounding box and the real bounding box, as well as the shape similarity between the predicted bounding box and the real bounding box are obtained, and based on the intersection-over-union ratio and the shape similarity, the loss value between the predicted bounding box and the real bounding box is determined; the initial defect detection model is trained with reducing the loss value as the training goal to obtain a trained defect detection model.

[0151] Each module in the aforementioned deep learning-based metal surface micro-defect detection system can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a computer device's memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0153] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A metal surface micro-defect detection method based on deep learning, characterized in that: The method comprises: Acquire the surface image of the metal to be detected and perform preprocessing; Calling a trained defect detection model to perform defect detection on the surface image to obtain a bounding box of a defect area in the surface image; Determining the actual defect size of the metal to be inspected based on the size information of the bounding box; The trained defect detection model includes an attention combination module, an enhanced feature extraction module, and a four-branch stacking module; the attention combination module is used to extract multi-scale features of the surface image; the enhanced feature extraction module extracts deep local features of the surface image by expanding the receptive field; and the four-branch stacking module reduces the number of model parameters by optimizing the network structure. The four-branch stacking module includes two branches; In the first branch, the input content is processed by a 1×1 convolution block to obtain the first output; In the second branch, the input content is processed by a 1×1 convolution block to obtain a second output, the second output is processed by a 3×3 convolution block to obtain a third output, the second output and the third output are concatenated and input into a 1×1 convolution block to obtain a fourth output; the fourth output is processed by a 3×3 convolution block to obtain a fifth output, and the fifth output is further processed by a 3×3 convolution block to obtain a sixth output; The first output, the fourth output, the fifth output and the sixth output are spliced and input into a 1×1 convolution block for processing to obtain the output result of the four-branch stacking module.

2. The metal surface micro-defect detection method based on deep learning according to claim 1, characterized in that: The method of obtaining the surface image of the metal to be detected and performing preprocessing is specifically as follows: Use an ultrasonic scanning microscope equipped with a 300MHz high-frequency probe and pure distilled water as a coupling agent to perform C-Scan scanning with a step size of 1μm to obtain high-resolution metal surface images; The collected high-resolution images are cropped into images of the same size, and the cropped image data is cleaned, including removing images that are unqualified after cropping and contain noise introduced during the scanning process.

3. The metal surface micro-defect detection method based on deep learning according to claim 1, characterized in that: The trained defect detection model includes a backbone network, a neck network and a detection head connected in sequence; The attention combination module is provided in the backbone network and the neck network; The enhanced feature extraction module is provided in the neck network; The four-branch stacking module is arranged in the neck network.

4. The metal surface micro-defect detection method based on deep learning according to claim 2, characterized in that: The attention combination module includes a multi-scale attention submodule and a downsampling feature extraction submodule; The multi-scale attention submodule adopts a three-stage feature optimization mechanism to achieve efficient feature enhancement. First, a multi-granularity feature expression is established through intelligent reshaping and grouping processing of the channel dimension. Secondly, a parallel branch architecture is used to synchronously capture global contextual information, and adaptive reweighting of feature channels is achieved based on the channel attention mechanism. Finally, an innovative cross-dimensional feature aggregation strategy is used to deeply fuse the multi-scale features output by each branch, thereby constructing an enhanced feature representation with rich semantic information.

5. The metal surface micro-defect detection method based on deep learning according to claim 2, characterized in that: The enhanced feature extraction module is obtained by adjusting the spatial pyramid pooling cross-stage partial connection module through the spatial pyramid fast pooling principle; and is used to extract deep local features through multi-scale pooling and cross-stage connection.

6. The metal surface micro-defect detection method based on deep learning according to claim 4 is characterized in that: The enhanced feature extraction module includes two branches. In the first branch, the input content is processed by multiple convolution blocks to obtain a first output. The first output is subjected to three consecutive pooling operations. The first output is concatenated with the output results of each pooling operation and then processed by two convolution blocks to obtain the first branch result. In the second branch, the input content is processed by a convolution block to obtain a second branch result; After the first branch result and the second branch result are spliced together, they are processed by a convolution block to obtain the output result of the enhanced feature extraction module.

7. The metal surface micro-defect detection method based on deep learning according to claim 2, characterized in that: The four-branch stacking module optimizes the network structure by adjusting the number and distribution of 1×1 convolution and 3×3 convolution, and reducing the splicing branches.

8. The metal surface micro-defect detection method based on deep learning according to claim 1, characterized in that: Determining the actual defect size of the metal to be inspected based on the size information of the bounding box specifically includes: Obtaining predetermined sizing equations; The size equation is obtained by fitting the actual defect size of the sample surface image with the size information of the bounding box of the defect area in the sample surface image; Substitute the size information of the bounding box into the size equation to obtain the actual defect size of the metal to be inspected; Defects are graded based on the length of their longest side: Draw a scatter plot based on the diagonal length of the predicted bounding box of several sample points and the longest side length of the actual defect to confirm their linear correlation; The least squares method is then used to obtain a linear fitting equation as the size equation. The size information of the bounding box obtained from subsequent actual inspection is further substituted into the size equation to obtain the actual defect size of the metal to be inspected.

9. The metal surface micro-defect detection method based on deep learning according to claim 1, characterized in that: The loss function for training defect detection is as follows: （8） Among them, IoU represents the intersection-over-union ratio between the predicted bounding box and the true bounding box. 、 Represents the shape similarity between the predicted bounding box and the true bounding box.

10. A metal surface micro-defect detection system based on deep learning, characterized in that: The system comprises: An acquisition unit, which scans the metal surface to be detected with an ultrasonic scanning microscope, obtains a surface image of the metal to be detected and performs preprocessing; The detection unit calls a defect detection model trained on a computer device to perform defect detection on the surface image to obtain a bounding box of a defect area in the surface image; wherein the trained defect detection model includes an attention combination module, an enhanced feature extraction module, and a four-branch stacking module; the attention combination module is used to extract multi-scale features of the surface image; the enhanced feature extraction module extracts deep local features of the surface image by expanding the receptive field; and the four-branch stacking module reduces the number of model parameters by optimizing the network structure; a determining unit, configured to determine an actual defect size of the metal to be detected based on the size information of the bounding box; The four-branch stacking module includes two branches; In the first branch, the input content is processed by a 1×1 convolution block to obtain the first output; In the second branch, the input content is processed by a 1×1 convolution block to obtain a second output, the second output is processed by a 3×3 convolution block to obtain a third output, the second output and the third output are concatenated and input into a 1×1 convolution block to obtain a fourth output; the fourth output is processed by a 3×3 convolution block to obtain a fifth output, and the fifth output is further processed by a 3×3 convolution block to obtain a sixth output; The first output, the fourth output, the fifth output and the sixth output are spliced and input into a 1×1 convolution block for processing to obtain the output result of the four-branch stacking module.

Images