A method for detecting surface defects of automobile sheet metal parts based on deep transfer learning

Abstract

The application discloses a kind of based on deep migration learning's automobile sheet metal surface defect detection method, comprising the following steps:1, image acquisition;2, image pre-processing, feature extraction;3, first-order network training;4, first-order network output, performance analysis;5, adjust sample, second-order network training;6, input image to be detected;7, network decision;8, reconstruct slice image, locate defect;9, detection result output.In the application, based on image processing, machine vision, deep learning, effectively detect automobile sheet metal surface defect through the above steps.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of defect detection, and particularly relates to a method for detecting surface defects of automobile sheet metal parts based on deep transfer learning. BACKGROUND

[0002] With the development of information technology, visual detection systems based on pattern recognition algorithms are increasingly applied in various industrial production activities instead of manual inspection. The automobile industry is a benchmark of high-end manufacturing industry in a country, and has high requirements for product quality and stability. Any tiny defect in the sheet metal parts of a vehicle body, such as scratches, scuffs, concave-convex damage, cracks, etc., may have a huge impact on the final product quality. Therefore, automatic identification and detection of defects on the surface of the sheet metal parts of the vehicle body is one of the most important processes in the automobile production and manufacturing process.

[0003] Traditional machine vision detection methods mainly include detection techniques based on bottom-layer pixel information and detection techniques based on machine learning algorithms. The detection techniques based on bottom-layer pixel information mainly identify edge, shape, area, HOG (Histogram of Oriented Gradient) and other features in view of the differences in pixel gray value between the defect area and the surrounding area. However, the changes in objective factors in the production environment can easily lead to great changes in pixel gray value, so this kind of method is only suitable for small-scale detection. The detection techniques based on machine learning algorithms mainly use SVM (Support Vector Machine), RBF (Radial Basis Function) and MLP (Multilayer Perceptron) algorithms, but this kind of method is difficult to capture global and high-order semantic features of images, so it is only suitable for simple detection tasks and has limited efficiency.

[0004] In recent years, with the rapid development of deep learning, researchers have begun to focus on using convolutional neural networks (CNN) to solve the problem of defect detection. However, the current defect detection method based on convolutional network mainly has the following problems: first, the object faced by the existing algorithm is usually an independent image reflecting an independent scene. Whether using the sliding window detection strategy of AlexNet or the target detection strategy of Faster R-CNN, once the network makes a mistake or misses a judgment, it will directly affect the detection result. That is, in one calculation, the single-stage network has a low fault tolerance rate for a single image. Second, the existing algorithm mainly uses random cropping, flipping, mirroring, contrast, and histogram adjustment for image preprocessing, which cannot adaptively weigh the relationship between the detection subject and the background area, resulting in the network sometimes learning irrelevant content and affecting the detection efficiency. Third, the existing algorithm's filtering and denoising method will affect both the detection subject and the noise, which will cause some detailed information of the detection subject to be filtered out, and these information will affect whether the network can learn effective features. In addition, the detection accuracy of the surface defects of the automobile industry is higher than that of the general industry, and the existing algorithm is difficult to meet the requirements. At the same time, deep learning is a big data driven technology, and the data samples provided by the industrial scene are relatively small, so directly training the network with small samples will easily cause overfitting. SUMMARY

[0005] The purpose of the present application is to provide a deep transfer learning-based automobile sheet metal surface defect detection method that can effectively solve the problems raised by the background technology.

[0006] To achieve the above purpose, the present application provides the following technical solution: a deep transfer learning-based automobile sheet metal surface defect detection method, comprising the following steps:

[0007] S1: Use an infrared thermal imager to perform tomographic scanning on the sheet metal part. For a product, a single scan can obtain hundreds of slice images. Experienced detection personnel label the collected images according to whether there are defects in the slice images, and construct an overall data set. A discernible defect usually exists in more than five slices, so even if a subsequent image is misjudged, the data acquisition method of the present application will make the misjudgment have a lower impact on the overall detection result.

[0008] S2: According to the detection subject, adaptively select the ROI (Region of Interest) of the image, segment, filter, and obtain a filtered image. Fuse the filtered image with the ROI image to obtain a sample image that filters out high-frequency noise and retains the details of the sheet metal part subject. Adjust the size and sample size of the sample image to make the data set required for the first-order network.

[0009] S3: Build a first-order network model, including two sub-networks, and train and verify the first-order model using the first-order training set and the first-order verification set.

[0010] S4: Save the training model of the first-order network and analyze the performance of each sub-network using the first-order test set.

[0011] S5: Label all FN (False Negative) samples output by the first-order two sub-networks as second-order positive samples, and label all FP (False Positive) samples output by the first-order two networks as second-order negative samples. Re-perform mirror image, random flip, etc. to expand the sample size, so that the positive and negative sample sizes are equal, both being 10,000 images, and build a second-order training set. Train and verify the second-order network using the second-order training set and the aforementioned second-order verification set.

[0012] S6: In the actual industrial detection process, the input data is the sheet metal part slice image directly collected by the infrared thermal imager without processing. The algorithm first performs ROI region selection, cropping, segmentation, and filtering on it, then adjusts its size to 512x512, and sends it to the network for calculation in order.

[0013] S7: The network calculates the input sample and outputs a discrimination value. Set the discrimination threshold. If the discrimination value is greater than or equal to 0.5, mark the slice as positive, otherwise mark the slice as negative.

[0014] S8: Reconstruct the sheet metal part slice directly collected by the infrared thermal imager without processing in order to obtain a two-dimensional reconstruction image that can reflect the true appearance of the sheet metal part, and mark the defect position on the reconstruction image.

[0015] S9: Output the image marked with the defect position and reflecting the true appearance of the sheet metal part, i.e. complete the final defect detection.

[0016] Further, in step S1, the FLIR-A615 industrial camera is installed on the image acquisition module to perform image acquisition; the camera focal length and various parameters are adjusted to enable clear capture of image surface information, and the image resolution is 640x480; in step S1, the collected slice images are sequentially named in bmp format, and professional detection personnel mark the images according to whether there are defects in the slice images; the images with defects are marked as positive samples, and the images without defects are marked as negative samples. The positive and negative samples are saved in different folders.

[0017] Further, in step S2, the density centroid algorithm is used to calculate the global gray value of the input image to obtain an anchor point. The input image is cropped to 640x200 based on the image length and the anchor point corresponding to the upper and lower 100 pixels, to obtain a cropped image. Since the pixel value of the sheet metal body is high in infrared imaging, and the pixel value of the background area is low, this step can obtain the ROI area of the sheet metal surface, and the irrelevant areas in the image are cropped, thereby improving the subsequent efficiency. The centroid calculation formula is:

[0018]

[0019] wherein x0 is the horizontal coordinate corresponding to the anchor point in the image coordinate system, y0 is the vertical coordinate corresponding to the anchor point, f ij is the gray value of the pixel at (x i , y j ),

[0020] In step S2, the ROI area obtained is subjected to threshold segmentation, and the pixel points with a gray value lower than 75 in the image are set to 0, and the gray values of the remaining pixel points remain unchanged, to further reduce the influence of the background area on the subsequent calculation, to obtain a segmented image,

[0021] In step S2, the segmented image obtained is subjected to Gaussian filtering with a kernel size of 49 to filter out high-frequency noise and irrelevant details in the image, to obtain a filtered image.

[0022] In step S2, the filtered image is fused with the ROI image to obtain a fused image, which retains the details of the detection subject in the original input image and filters out irrelevant background areas and high-frequency noise. The fusion formula is:

[0023] I SPL =I ROI ⊙I GS

[0024] wherein I SPL represents the fused image, I ROI is the ROI image, and I GSis a Gaussian filtered image, and ⊙ is the Hadamard matrix product.

[0025] In step S2, the fused image is resized using bilinear interpolation from 640×200 to 525×525. Then, the resized image is randomly cropped to 512×512, and mirroring and rotation operations are used to expand the sample size, ensuring that the number of positive and negative samples are equal, both at 24,000 images. Next, 10,000 images from each of the positive and negative samples are used as the training set for the first-order network, 2,000 images from each are used as the validation set, 10,000 images from each are used as the test set, and 2,000 images from each are used as the validation set for the second-order network, thus constructing the dataset.

[0026] Furthermore, in step S3, the first-order model is constructed in parallel by two independent networks, one using a VGG19 structure and the other a ResNet50 structure, and each network is trained and validated on samples separately. The VGG19 network contains 16 convolutional layers and 3 fully connected layers, using 3×3 convolutional kernels and 2×2 max-pooling layers from input to output. For a given receptive field, compared to networks like AlexNet that use large convolutional kernels, VGG19's strategy of stacking small convolutional kernels allows it to utilize more non-linear layers to learn more complex patterns with lower cost and fewer parameters. The ResNet50 network contains 4 residual blocks and 50 convolutional layers. Shortcut connections are set within the residual blocks, allowing feature maps from the earlier layers to skip multiple layers and be cascaded to later convolutional layers. This aims to overcome problems such as significant gradient vanishing and lower learning rates caused by increased depth, and to improve accuracy. This invention employs a transfer learning strategy, and the VGG19 and ResNet50 backbones used have already undergone parameter initialization and pre-training on the large-scale ImageNet dataset. Furthermore, to make the network more suitable for image classification tasks, this invention freezes the original output layers of both networks and defines new fully connected layers as replacements. The input layer of the fully connected layer has a mapping size of 512×512 to accommodate large input sizes, the activation function is ReLU (Rectified Linear Unit), and DropOut is set to 0.5. The output layer activation function is the Sigmoid function, and the output is a real number in the range of 0 to 1.

[0027] In step S3, the network is trained using a stochastic gradient descent strategy. The input sample size is 512×512, the batch size is set to 1, the initial learning rate is set to 1×10⁻⁴, the momentum is set to 0.9, the data traversal is one round, and the training rounds are set to 20. After each round of training, the trained model is saved based on the validation results to suppress overfitting. The loss function is calculated as follows:

[0028]

[0029] Where img represents the input sample; f(·) represents the network operation process; lab represents the sample label, when the sample is positive, lab=1, when the sample is negative, lab=0; m represents the number of samples.

[0030] Furthermore, in step S4, the trained VGG19 network and ResNet50 network are tested using the first-order test set. For each input image, the network provides a predicted value. If the value is greater than or equal to 0.5, the input is considered a positive sample; otherwise, it is considered a negative sample. The test results of the first-order model can be divided into four categories: TP (True Positive), where a positive sample is predicted as a positive sample; TN (True Negative), where a negative sample is predicted as a negative sample; FP (False Positive), where a negative sample is predicted as a positive sample; and FN (False Negative), where a positive sample is predicted as a negative sample. Based on these four results, this invention uses precision and recall metrics to evaluate the two networks. The higher the evaluation value, the better the network performance. The evaluation formula F is:

[0031] F = 2PR / (P + R)

[0032] Where P represents accuracy, and its calculation formula is:

[0033] P = TP / (TP + FP)

[0034] R is the recall rate, and its calculation formula is:

[0035] P = TP / (TP+FN) In step S4, the higher-performing network is selected and its model is used as the backbone of the second-order network. This invention is based on a transfer learning strategy, using the training weights of the first-order network as the initial parameters of the second-order network.

[0036] Furthermore, in step S5, the network is trained using a stochastic gradient descent strategy, with 10,000 positive and 10,000 negative images. The input sample size is 512×512, the batch size is set to 1, the initial learning rate is set to 1×10⁻⁴, the momentum is set to 0.9, the data is traversed in one round, and the training rounds are set to 10. After each round of training, the trained model is saved based on whether the validation results show improvement, in order to suppress overfitting. The loss function is the same as that of the first-order model.

[0037] In step S5, after the training of the second-order network is completed, the algorithm is encapsulated, and the model weights adopt the training weights of the second-order network.

[0038] Furthermore, in step S7, for actual defects existing in the sheet metal parts, considering factors such as camera resolution and defect identification size, a defect typically exists in five or more consecutive slices. Therefore, if a slice is marked as positive during the network discrimination process, and the two consecutive slices above and below it are both marked as negative, the algorithm considers it a false positive and re-marks it as negative; slices marked as negative during the network discrimination process are also re-marked in the same way. This can further reduce the algorithm's false positive and false negative rates. Because false positives and false negatives are usually independent and occasional, the probability of multiple consecutive slices being misjudged simultaneously is low.

[0039] In step S7, after re-marking the slices, the algorithm further marks continuous positive slices as defective regions and negative continuous slices as defect-free regions, thus completing the detection process.

[0040] Furthermore, in step S8, the Region of Interest (ROI) is selected for the first image of a batch of input slices, and subsequent slices are cropped based on the range corresponding to this region. Then, the calculation is performed on each slice individually. For a single slice, a weighted average is calculated row by row. A 640×200 ROI slice will become a 640×1 array after this operation, which represents one row of the reconstructed image. The weighted average formula is:

[0041]

[0042] Among them, f ij f represents the original grayscale value of the pixel in the j-th row and i-th column of a slice. i ' is the weighted average value of this column, and row represents the total number of rows in this slice. Traversing the input slices yields the reconstructed image, and then the defect locations are marked on the reconstructed image.

[0043] This invention provides a method for detecting surface defects in automotive sheet metal parts based on deep transfer learning, which has the following advantages:

[0044] 1. To overcome the problem of insufficient sample size required for deep network training, this invention uses infrared thermal tomography instead of traditional industrial area array cameras to acquire a series of slice images of the sheet metal surface. This method can provide sufficiently deep convolutional neural networks with diverse samples sufficient for learning multi-order, deep nonlinear complex mapping relationships, enabling the model to more fully fit the data features and reduce the overfitting rate.

[0045] 2. To avoid having too many regions in the samples that are irrelevant to the main body of the sheet metal part to be detected, which would cause the deep network to perform too many unnecessary feature fittings and consume computing resources, this invention designs an image processing flow for the main body of the sheet metal part. Through operations such as centroid calculation, adaptive main body region selection, image cropping, and threshold segmentation, irrelevant regions are removed as much as possible to improve the detection efficiency of the algorithm.

[0046] 3. Traditional deep networks use convolutional filtering and sampling methods that simultaneously affect the detected subject and high-frequency noise. To protect the features of the sheet metal part, especially its detailed information, from being destroyed by global filtering, this invention designs a novel feature fusion sample to replace the traditional original image sample input to the network for training, verification, and testing. This sample can simultaneously highlight the features of the detected subject and eliminate high-frequency noise, allowing the network to focus more on learning the features of the sheet metal part.

[0047] 4. To enhance the feature fitting ability of deep networks, this invention employs a transfer learning strategy. The weights used in the first-order network are derived from the large-scale ImageNet dataset. Although the sheet metal detection task addressed in this invention differs somewhat from the scenarios addressed by ImageNet, the effective visual feature space hierarchy learned by deep networks across different problems is transferable. Furthermore, the weights used in the second-order network are transferred from the training model of the first-order network, progressively improving the network's learning ability.

[0048] 5. To improve the accuracy of the algorithm, this invention adopts the Boosting idea and reinforcement learning strategy, cascading weak classifiers to build strong classifiers. Based on the performance analysis and evaluation of the network based on the detection results of the first-order network, the training samples of the second-order network are actively and specifically adjusted to correct the weight parameters that cause false detections in the deep network, thereby further improving the robustness of the network.

[0049] 6. To improve the algorithm's fault tolerance, this invention, based on the defect characteristics and detection requirements of the automotive sheet metal parts it targets, performs secondary annotation on the final discrimination results at the slice level using deep network discriminant values. Due to the unique data acquisition method of this invention, a defect may exist simultaneously and continuously in multiple slices. However, false positives and false negatives are usually isolated and occasional. Therefore, this secondary annotation for occasional cases can improve the algorithm's fault tolerance. Attached Figure Description

[0050] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention, but do not constitute a limitation thereof. In the drawings:

[0051] Fig. 1 This is a flowchart of the method described in this invention;

[0052] Fig. 2 A network structure diagram of the method described in the invention; Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figs. 1-2 This invention provides a technical solution: a method for detecting surface defects in automotive sheet metal parts based on deep transfer learning, comprising the following steps:

[0055] S1: Use an infrared thermal imager to perform tomographic scanning on sheet metal parts. For a single product, a single scan can obtain hundreds of slice images. Experienced inspectors can label the acquired images based on whether there are defects in the slice images and construct an overall dataset. A identifiable defect usually exists in more than five slices. Therefore, even if a subsequent image is misjudged, the data acquisition method of this invention will minimize the impact of misjudgment on the overall inspection results.

[0056] S2: Based on the detected subject, adaptively select, segment, and filter the ROI (Region of Interest) of the image to obtain a filtered image. Fuse the filtered image with the ROI image to obtain a sample image that removes high-frequency noise while preserving the details of the sheet metal part. Adjust the size and scale of the sample images to create the dataset required for the first-order network.

[0057] S3: Construct a first-order network model, which contains two sub-networks. Train and validate the first-order model using the first-order training set and the first-order validation set.

[0058] S4: Save the training model of the first-order network and use the first-order test set to perform performance analysis on each sub-network.

[0059] S5: Label all FN (False Negative) samples from the test outputs of the two sub-networks of the first stage as positive samples of the second stage, and label all FP (False Positive) samples from the test outputs of the two sub-networks of the first stage as negative samples of the second stage. Then, perform mirroring and random flipping operations to expand the sample size, making the number of positive and negative samples equal at 10,000 images each, and construct the second-stage training set. Use the second-stage training set and the aforementioned second-stage validation set to train and validate the second-stage network.

[0060] S6: In actual industrial inspection, the input data is unprocessed slice images of sheet metal parts directly acquired by an infrared thermal imager. The algorithm first selects, crops, segments, and filters the ROI region, then adjusts its size to 512×512, and sends it sequentially into the network for calculation.

[0061] S7: The network calculates the input sample and outputs the discriminant value. It sets the discriminant threshold. If the discriminant value is greater than or equal to 0.5, the slice is marked as positive; otherwise, the slice is marked as negative.

[0062] S8: Reconstruct the unprocessed sheet metal slices directly acquired by the infrared thermal imager in sequence to obtain a two-dimensional reconstructed image that reflects the true appearance of the sheet metal, and mark the defect location on the reconstructed image.

[0063] S9: Outputs an image that marks the location of defects and reflects the true appearance of the sheet metal part, thus completing the final defect detection.

[0064] In this embodiment, in step S1, a FLIR-A615 industrial camera is mounted on the image acquisition module for image acquisition. The camera's focal length and other parameters are adjusted to clearly capture surface information, with an image resolution of 640×480. Simultaneously, in step S1, the acquired slice images are named sequentially in BMP format. Professional inspectors label the images based on the presence or absence of defects; images with defects are labeled as positive samples, and images without defects are labeled as negative samples. The positive and negative samples are saved in different folders.

[0065] In this embodiment, in step S2, the global grayscale value of the input image is calculated using the density centroid algorithm to obtain an anchor point. The input image is then cropped to 640×200 pixels, with the image length as the length and the width as 100 pixels above and below the anchor point. Because the sheet metal body has a high pixel value in infrared imaging, while the background area has a low pixel value, this step can obtain the ROI region for the sheet metal surface, while removing areas in the image irrelevant to the calculation, thereby improving subsequent efficiency. The centroid calculation formula is:

[0066]

[0067] Where x0 is the x-coordinate of the anchor point in the image coordinate system, y0 is the y-coordinate of the anchor point, and f ij For pixels in (x i y j The grayscale value at )

[0068] In step S2, threshold segmentation is performed on the acquired ROI region. Pixels with a grayscale value below 75 are set to 0, while the grayscale values ​​of the remaining pixels remain unchanged. This further reduces the influence of the background region on subsequent calculations, resulting in a segmented image.

[0069] In step S2, the obtained segmented image is subjected to Gaussian filtering with a kernel size of 49 to filter out high-frequency noise and irrelevant details in the image, resulting in a filtered image.

[0070] In step S2, the filtered image and the ROI image are fused to obtain a fused image. The fused image retains the details of the detected subject in the original input image while filtering out irrelevant background areas and high-frequency noise. The fusion formula is:

[0071] I SPL =I ROI ⊙I GS

[0072] Among them, I SPL Indicates the fused image, I ROI For the ROI image, I GS is a Gaussian filtered image, and ⊙ is the Hadamard matrix product.

[0073] In step S2, the fused image is resized using bilinear interpolation from 640×200 to 525×525. Then, the resized image is randomly cropped to 512×512, and mirroring and rotation operations are used to expand the sample size, ensuring that the number of positive and negative samples are equal, both at 24,000 images. Next, 10,000 images from each of the positive and negative samples are used as the training set for the first-order network, 2,000 images from each are used as the validation set, 10,000 images from each are used as the test set, and 2,000 images from each are used as the validation set for the second-order network, thus constructing the dataset.

[0074] In this embodiment, in step S3, the first-order model is constructed in parallel by two independent networks. The two networks use VGG19 and ResNet50 structures as their backbones, respectively, and are trained and validated on samples separately. The VGG19 network contains 16 convolutional layers and 3 fully connected layers, using 3×3 convolutional kernels and 2×2 max-pooling layers from input to output. For a given receptive field, compared to networks like AlexNet that use large convolutional kernels, VGG19's strategy of stacking small convolutional kernels allows it to utilize more non-linear layers to learn more complex patterns with lower cost and fewer parameters. The ResNet50 network contains 4 residual blocks and 50 convolutional layers. Shortcut connections are set within the residual blocks, allowing feature maps from the earlier layers to skip multiple layers and be cascaded to later convolutional layers. This aims to overcome problems such as significant gradient vanishing and lower learning rates caused by increased depth, and improve accuracy. This invention employs a transfer learning strategy, and the VGG19 and ResNet50 backbones used have already undergone parameter initialization and pre-training on the large-scale ImageNet dataset. Furthermore, to make the network more suitable for image classification tasks, this invention freezes the original output layers of both networks and defines new fully connected layers as replacements. The input layer of the fully connected layer has a mapping size of 512×512 to accommodate large input sizes, the activation function is ReLU (Rectified Linear Unit), and DropOut is set to 0.5. The output layer activation function is the Sigmoid function, and the output is a real number in the range of 0 to 1.

[0075] In step S3, the network is trained using a stochastic gradient descent strategy. The input sample size is 512×512, the batch size is set to 1, the initial learning rate is set to 1×10⁻⁴, the momentum is set to 0.9, the data traversal is one round, and the training rounds are set to 20. After each round of training, the trained model is saved based on the validation results to suppress overfitting. The loss function is calculated as follows:

[0076]

[0077] Where img represents the input sample; f(·) represents the network operation process; lab represents the sample label, when the sample is positive, lab=1, when the sample is negative, lab=0; m represents the number of samples.

[0078] In this embodiment, in step S4, the trained VGG19 network and ResNet50 network are tested using the first-order test set. For each input image, the network provides a predicted value. If the value is greater than or equal to 0.5, the input is considered a positive sample; otherwise, it is considered a negative sample. The test results of the first-order model can be divided into four categories: TP (True Positive), where a positive sample is predicted as a positive sample; TN (True Negative), where a negative sample is predicted as a negative sample; FP (False Positive), where a negative sample is predicted as a positive sample; and FN (False Negative), where a positive sample is predicted as a negative sample. Based on these four results, this invention uses precision and recall metrics to evaluate the two networks. The higher the evaluation value, the better the network performance. The evaluation formula F is:

[0079] F = 2PR / (P + R)

[0080] Where P represents accuracy, and its calculation formula is:

[0081] P = TP / (TP + FP)

[0082] R is the recall rate, and its calculation formula is:

[0083] P = TP / (TP + FN)

[0084] In step S4, the higher-performing of the two networks is selected, and its model is used as the backbone of the second-order network. This invention is based on a transfer learning strategy, using the training weights of the first-order network as the initial parameters of the second-order network.

[0085] In this embodiment, in step S5, the network is trained using a stochastic gradient descent strategy. The number of positive and negative samples is 10,000 images each, the input sample size is 512×512, the batch size is set to 1, the initial learning rate is set to 1×10⁻⁴, the momentum is set to 0.9, the data is traversed in one round, and the training rounds are set to 10. After each round of training, the trained model is saved based on whether the validation results show improvement, in order to suppress overfitting. The loss function is the same as that of the first-order model.

[0086] In step S5, after the training of the second-order network is completed, the algorithm is encapsulated, and the model weights adopt the training weights of the second-order network.

[0087] In this embodiment, in step S7, for real defects existing in actual sheet metal parts, considering factors such as camera resolution and defect identification size, a defect typically exists in five or more consecutive slices. Therefore, if a slice is marked as positive during the network discrimination process, and the two consecutive slices above and below it are both marked as negative, the algorithm considers it a false positive and re-marks it as negative; slices marked as negative during the network discrimination process are also re-marked in the same way. This can further reduce the algorithm's misjudgment and missed judgment rates. Because misjudgments and missed judgments are usually independent and occasional, the probability of multiple consecutive slices being misjudged simultaneously is low.

[0088] In step S7, after re-marking the slices, the algorithm further marks continuous positive slices as defective regions and negative continuous slices as defect-free regions, thus completing the detection process.

[0089] In this embodiment, in step S8, the Region of Interest (ROI) is selected for the first image of a batch of multiple input slices, and subsequent slices are cropped using the range corresponding to this region. Then, the calculation is performed on each slice individually. For a single slice, a weighted average is calculated row by row. A 640×200 ROI slice will become a 640×1 array after this operation, which represents one row of the reconstructed image. The weighted average formula is:

[0090]

[0091] Among them, f ij f′ represents the original grayscale value of the pixel in the j-th row and i-th column of a slice. i This is the weighted average value of the column, and row represents the total number of rows in the slice. Traversing the input slices yields the reconstructed image, and then the defect locations are marked on the reconstructed image.

[0092] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting surface defects in automotive sheet metal based on deep transfer learning, characterized in that: Includes the following steps: S1: Use an infrared thermal imager to perform tomographic scanning on sheet metal parts. For a single product, a single scan can obtain hundreds of slice images. Experienced inspectors can annotate the acquired images based on whether there are defects in the slice images, and construct an overall dataset. A identifiable defect may exist in more than five slices. S2: Based on the detected subject, adaptively select, segment, and filter the Region of Interest (ROI) of the image. Specifically, the global grayscale value of the input image is calculated using the centroid density algorithm to obtain an anchor point. The input image is then cropped to 640×200 pixels, with the image length as the length and the width as 100 pixels above and below the anchor point. Since the sheet metal part has a high pixel value in infrared imaging, while the background area has a low pixel value, this step can obtain the ROI region for the sheet metal part surface, while removing areas in the image irrelevant to the calculation, thus improving the efficiency of subsequent steps. The centroid calculation formula is: in, The x-coordinate of the anchor point in the image coordinate system. The vertical coordinate corresponding to the anchor point. For pixels in grayscale value at that location Next, threshold segmentation is performed on the acquired ROI region. Pixels with a gray value below 75 are set to 0, while the gray values ​​of other pixels remain unchanged to further reduce the influence of the background region on subsequent calculations, resulting in a segmented image. This segmented image is then subjected to a Gaussian filter with a kernel size of 49 to remove high-frequency noise and irrelevant details, resulting in a filtered image. This filtered image is then fused with the ROI image to obtain a sample image that removes high-frequency noise while retaining the details of the sheet metal part. The fusion formula is as follows: in, Indicates a fused image. For ROI images, The image is a Gaussian filtered image. This is the product of Hadamard matrices; Adjust the size and sample scale of the sample images to create the dataset required for the first-order network; S3: Construct a first-order network model, which includes two sub-networks. Train and validate the first-order model using the first-order training set and the first-order validation set. The first-order model consists of two independent networks in parallel. The two networks are based on VGG19 and ResNet50 structures, respectively, and are trained and validated on samples. The VGG19 and ResNet50 backbones have been initialized and pre-trained on the ImageNet large-scale dataset. In addition, to make the network more suitable for image classification tasks, the original output layers of the two networks are frozen, and a new fully connected layer is defined as a replacement. The input layer of the fully connected layer is mapped to a size of 512×512 to accommodate large input sizes. The activation function is ReLU, and DropOut is set to 0.

5. The output layer activation function is the Sigmoid function, and the output is a real number in the range of 0 to 1. S4: Save the training model of the first-order network, and use the first-order test set to perform performance analysis on each sub-network. Select the one with higher performance among the two sub-networks and use its model as the backbone of the second-order network. S5: Label all false negative FN samples from the test output of the two sub-networks of the first order as positive samples of the second order, and label all false positive FP samples from the test output of the two sub-networks of the first order as negative samples of the second order. Re-mirror and randomly flip them to expand the sample size so that the number of positive and negative samples is equal, both being 10,000 images. Construct a second order training set. Use the second order training set and the second order validation set to train and validate the second order network. After the training of the second order network is completed, encapsulate it. The model weights used in actual industrial detection adopt the training weights of the second order network. S6: In the actual industrial inspection process, the input data is the unprocessed sheet metal slice image directly acquired by the infrared thermal imager. First, the ROI region is selected, cropped, segmented and filtered. Then, its size is adjusted to 512×512 and sent into the network model for calculation in sequence. S7: The network calculates the input sample and outputs the discriminant value. It sets the discriminant threshold. If the discriminant value is greater than or equal to 0.5, the slice is marked as positive; otherwise, the slice is marked as negative. S8: Reconstruct the unprocessed sheet metal slices directly acquired by the infrared thermal imager in sequence to obtain a two-dimensional reconstructed image that reflects the true appearance of the sheet metal, and mark the defect location on the reconstructed image. S9: Outputs an image that marks the location of defects and reflects the true appearance of the sheet metal part, thus completing the final defect detection.

Images