A workpiece defect detection method based on a deep neural network

Abstract

The application provides a workpiece defect detection method based on a deep neural network. Image enhancement processing is performed on training images according to global features of the images, thereby reducing the problem of different image qualities caused by different environmental illuminations and enhancing the environmental adaptability of a defect detection algorithm. The deep neural network is used as a backbone network for feature extraction, so that deeper semantic information is obtained. Feature fusion of shallow features and deep information is performed in a feature pyramid, context information is fully utilized, and an attention mechanism is introduced to suppress the interference of the fused features on original detailed features, so that effective fusion of semantic information and detailed information is realized, and the detection performance for defects of various scales is improved. The application can accurately and effectively detect defects of various types and various scales in product images under different illumination conditions in a component data set of a workpiece.

Description

Technical Field

[0001] This invention relates to a method for detecting defects in workpieces, and more particularly to a method for detecting defects in workpieces based on deep neural networks. Background Technology

[0002] The development trend of global manufacturing is closely related to intelligentization. Intelligent manufacturing and smart factories are important development trends in global manufacturing. By promoting the formation of new production management methods, business operation models, and industrial development forms, they will have a significant impact on the global industrial landscape. Complex component manufacturing technology has wide applications in manufacturing. Among these, automatic workpiece defect detection algorithms are an important part of improving workpiece production efficiency, and solving this key problem has very important practical value.

[0003] Defect detection is of paramount importance in industrial product manufacturing. During production, due to factors such as processing, design, machine tool malfunctions, and harsh operating conditions, manufactured products inevitably contain various defects such as holes, dents, and scratches. In daily use, these defects can easily lead to corrosion and fatigue damage, increasing production costs, causing significant resource waste and economic losses, and even posing serious risks to human safety. With the continuous development of the equipment manufacturing industry, the demand for high-standard, high-quality products in high-end manufacturing equipment is increasing, placing higher demands on the skills of designers and on-site inspection personnel. Therefore, in complex industrial processes, accurate and effective defect detection of manufactured products helps improve product quality and enhance the core competitiveness of manufacturing enterprises.

[0004] In factory production, manual inspection often involves on-site visual inspection, aided by various inspection tools. Manual inspection has certain limitations: low efficiency, inability to objectively guarantee inspection quality, high skill requirements for inspectors, and high resuscitation costs. Automated defect detection technology offers significant advantages over manual inspection. It can adapt to harsh environments, operate for extended periods with high precision and efficiency, and provides specific numerical values ​​for the reliability of inspection results. The technology of using machine vision equipment to acquire images and determine the presence of defects is becoming a widespread industrial demand. Currently, automated defect detection equipment based on machine vision is gradually replacing manual visual inspection in various industrial sectors, including 3C, automotive, home appliance, machinery manufacturing, semiconductor and electronics, chemical, pharmaceutical, aerospace, and light industry. Automated defect detection offers significant advantages over traditional manual inspection, reducing labor costs and improving product quality and production efficiency.

[0005] Automatic detection of workpiece defects presents several challenges. For example, significant structural differences exist between different workpiece models, or different types of defects exhibit vastly different characteristics. This makes it difficult to apply the same algorithm to detect defects in multiple workpieces. Furthermore, the same defect may vary in size or have random locations, hindering the algorithm's ability to achieve ideal accuracy in practical applications. Traditional defect detection methods typically employ conventional image processing algorithms or manually designed feature classifiers. By utilizing the different properties of the inspected surface or defect, appropriate imaging schemes are designed. A suitable imaging scheme helps obtain images with uniform or specific lighting conditions, thus highlighting surface defects. In real and complex industrial environments, defect detection often faces many challenges, such as small differences between defect images and backgrounds, low contrast, large variations in defect scale and diverse types, a large amount of noise in product images, or large interference due to defects under certain special lighting conditions. In such cases, traditional methods often seem helpless and difficult to achieve the desired results. For example: (1) In industrial environments, when some new surface defects appear, traditional algorithms need to design new solutions based on their characteristics; (2) Surface defects may be "uncertain" in size, shape and location, and using a standard single feature descriptor for detection will lead to insufficient classification results; (3) When the imaging environment is poor and the image contains a lot of noise, even if traditional algorithms take image enhancement into account, their improvement in accuracy is very limited. Summary of the Invention

[0006] Traditional defect detection algorithms have limitations, while deep learning algorithms can automatically learn sample features, thus offering performance advantages over some traditional algorithms in image processing. In recent years, deep learning models, represented by convolutional neural networks, have been successfully applied in many computer vision fields, such as face recognition, pedestrian recognition, scene text detection, object tracking, and autonomous driving. Many deep learning-based defect detection methods are also widely used in various industrial scenarios. Deep learning-based workpiece defect detection methods not only have significant academic research value but also possess very broad market application prospects.

[0007] To address the above problems, based on existing image acquisition and target detection models, and combined with deep learning methods, a workpiece defect detection algorithm based on a deep neural network is designed. Image enhancement processing is performed on the training images according to the global features of the images to mitigate the problem of inconsistent image quality caused by different ambient lighting conditions, thus enhancing the environmental adaptability of the defect detection algorithm. The deep neural network is used as the backbone network for feature extraction to obtain deeper semantic information. Feature fusion of shallow and deep information is performed in the feature pyramid, making full use of contextual information, and an attention mechanism is introduced to suppress the interference of fused features on the original detailed features, achieving effective fusion of semantic and detailed information and improving the detection performance for defects at various scales. This invention can accurately and effectively detect defects of various types and scales in product images under different lighting conditions in a workpiece component dataset. The main contents include the following aspects:

[0008] 1. Randomly crop, rotate, and scale the acquired workpiece images, and randomly change the image brightness and contrast to generate a workpiece image dataset. Then, use a simple neural network to predict the hyperparameters of the image enhancement module. Based on the predicted hyperparameters, perform image enhancement in a specific direction on the training images in the differentiable image processing module.

[0009] 2. By introducing an attention mechanism and a feature fusion module into the deep neural network, the network can focus more on fusing effective features between channels during training. This integrates shallow features with deep semantic information, enabling shallow feature maps used for small-scale target detection and localization to simultaneously contain details and deep semantic information, thus achieving accurate detection of defects at multiple scales.

[0010] 3. Using the enhanced image and features obtained through the feature fusion module as input vectors, and the defect location and defect category as output vectors, deep neural networks are used for deep learning to fit a deep model for defect detection.

[0011] This invention provides a workpiece defect detection method based on deep neural networks, comprising the following steps:

[0012] Step 1: Preprocess the acquired workpiece images to generate a workpiece image dataset, and perform image enhancement on the training images in the workpiece image dataset.

[0013] Step 2 involves introducing an attention mechanism and a feature fusion module into the deep neural network. Deep learning is then used to fit a deep model for defect detection, and the output defect location and defect category are used as the final result of workpiece defect detection. This specifically includes the following sub-steps:

[0014] Step 2.1: The enhanced image from Step 1 is used as input and trained and learned through the backbone network of a deep neural network, which is used to extract image features;

[0015] Step 2.2: The image features extracted by the backbone network are used to construct feature maps of different scales through a series of convolutional layers, and the multi-scale feature map part is denoted as the feature pyramid.

[0016] Step 2.3: Add a feature fusion module to the deep neural network to fuse shallow features with deep semantic information. That is, it is used to stitch together the image features extracted by the backbone network with the feature map in the feature pyramid so that the shallow feature map used for small-scale target detection and localization can contain both details and deep semantic information.

[0017] Step 2.4: Defect detection is performed using the feature pyramid obtained by feature fusion in the deep neural network.

[0018] Furthermore, in step 1, an image enhancement module is constructed, consisting of a parameter prediction module and a differentiable image processing module, and the training images are enhanced using the image enhancement module.

[0019] The parameter prediction module includes five convolutional layers and two fully connected layers. Each convolutional layer is followed by an activation layer. The input is a downsampled low-resolution image, and the output is the hyperparameters required by the differentiable image processing module.

[0020] The differentiable image processing module enhances images using algorithms for brightness, contrast, and hue gamma correction, contrast transformation, and white balance transformation, as well as sharpening algorithms that enhance image edge and detail features.

[0021] Furthermore, in step 2.1, the backbone network is ResNet50.

[0022] Furthermore, in step 2.2, the feature map set output by the backbone network ResNet50 is denoted as {C1, C2, C3, C4}. The shallow feature layer C2 with a small scale is selected as the initial feature map of the feature pyramid, and the feature pyramid is constructed according to the following formula:

[0023] P i+1 =F(f 3×3 (F(f 1×1 (P i ))))

[0024] F(x i ) = ReLU(BN(x i ))

[0025] Where i∈{0, 1, 2, 3, 4}, P0=C2, f3×3 and f 1×1 These are convolutional layers with kernel sizes of 3 and 1, respectively, f 3×3 The stride of the convolutional layer is 2, BN is batch normalization, and F(x) i To activate the batch-standardized results, ReLU is the activation function. The final output feature map set is represented as {P0, P1, P2, P3, P4, P5}, with feature map sizes of {38, 19, 10, 5, 3, 1}.

[0026] Furthermore, in step 2.3, the feature fusion module can be represented by the following formula:

[0027] T f =Concat(f{X p},f{Up(X q )})

[0028] f(x i ) = ReLU(L2(x i ))

[0029] Up(x i ) = ReLU(BN(f 1×1 (Bilinear(x i )))

[0030] Where, x i Let f represent the i-th feature map to be fused, Bilinear represents bilinear interpolation, BN is batch normalization, ReLU is the activation function, and f 1×1 For convolutional layers with a kernel size of 1, for feature maps to be fused that differ in size from the target size, a 1×1 convolutional layer is first used to reduce the dimensionality. Then, bilinear interpolation is used to upsample the smaller deep feature maps. f represents the feature maps to be fused. L2 regularization is applied to normalize the data to the same range, facilitating subsequent concatenation. A ReLU activation function is applied after L2 regularization. The concatenation method is a cascade concatenation operation, which combines the unupsampled feature map X... p and the interpolated feature map X q The layers are concatenated along the channel dimension, and then a 3×3 convolutional layer is used to eliminate the aliasing effect after concatenation. f This is the result of fusing shallow features with deep semantic information.

[0031] Furthermore, step 2.3 introduces an attention mechanism, namely SE Block, which enables the network to focus more on fusing effective features between channels during training. The attention mechanism is introduced after L2 regularization + ReLU activation function and before concatenation processing.

[0032] Furthermore, the attention mechanism processes are as follows;

[0033] The input feature layer first obtains global spatial information through a global average pooling layer, and then compresses and increases the dimensionality of the global information through two fully connected layers and two activation functions. Finally, the weights of each channel are obtained and multiplied with the corresponding feature maps to obtain the correlation between channels, so that the network can focus more on fusing effective features between channels during training.

[0034] Furthermore, in step 2.4, dense sampling is performed on different locations of each feature map in the fused feature pyramid. Different bounding box scales and aspect ratios are used during sampling to perform object classification and prior box regression. The object classification result is output as the defect category, and the position of the prior box is output as the defect location.

[0035] Furthermore, the sharpening algorithm employs the USM algorithm.

[0036] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0037] 1. It has high adaptability to workpiece product inspection under different imaging conditions, light intensities, and image backgrounds in factories. In the actual workpiece production process, optical imaging systems are subject to many influences. Many existing methods can only be used to inspect product images with uniform backgrounds and clear workpieces acquired under ideal conditions. However, this invention, through a differentiable image processing module, can filter out image noise and improve image contrast, making the features of the target area to be inspected more prominent.

[0038] 2. It can also achieve relatively accurate detection even for workpiece datasets with small sample sizes and simple defect types. Currently, the datasets in related studies such as surface defect detection, PCB defect detection, and LCD screen defect detection are not abundant enough. In addition, in actual production, there are often many normal samples, while the number of defective workpieces is relatively small. This invention designs a defect dataset in the model with random cropping, rotation and scaling, and random changes to image brightness and contrast, which can improve the model's defect detection capability.

[0039] 3. The model of this invention has a certain degree of universality and can detect defects of different types and sizes. Deep neural networks often struggle to achieve stable and accurate detection rates for targets of different types or scales. This invention, by introducing an attention mechanism and a feature fusion module, fuses shallow features with deep semantic information, enabling accurate detection of defects at multiple scales. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the image enhancement module.

[0041] Figure 2 This is a schematic diagram of the attention mechanism and feature fusion structure.

[0042] Figure 3 This is a schematic diagram of a deep learning model and defect detection (i.e., the overall flowchart of the method). Detailed Implementation

[0043] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] like Figure 3 As shown, the present invention provides a workpiece defect detection method based on a deep neural network, comprising the following steps:

[0045] Step 1: Randomly crop, rotate, and scale the acquired workpiece images, and randomly change the image brightness and contrast to generate a workpiece image dataset. Then, use a simple neural network to predict the hyperparameters of the image enhancement module. Based on the predicted hyperparameters, perform image enhancement on the training images in the differentiable image processing module. The image enhancement module consists of two parts: a parameter prediction module and a differentiable image processing module.

[0046] Step 1-1: The neural network used for parameter prediction mainly consists of five convolutional layers and two fully connected layers. Each convolutional layer is followed by an activation layer. The input is a downsampled, low-resolution image, and the fully connected layers output the hyperparameters required by the differentiable image processing module. Based on global image features, such as hue and brightness, the neural network predicts the parameters for image filtering. Hyperparameters that would otherwise require manual adjustment are predicted by this simple neural network and continuously optimized and learned through backpropagation using the loss function. This allows the model to adaptively enhance the image in a direction favorable to detection.

[0047] Step 1-2: Based on the image features of the collected workpiece parts, the following image enhancement algorithms were selected: Gamma correction, contrast transformation, and white balance transformation algorithms to improve image brightness, contrast, and hue, as well as sharpening algorithms to enhance image edge and detail features. The various filter algorithms are shown in Table 1.

[0048] Table 1. Filter Algorithm for Differentiable Image Processing Module

[0049]

[0050] *P i r represents the pixel values ​​of the image. i g i b i The pixel values ​​at the corresponding positions of each channel

[0051] In this system, Gamma is used to correct the brightness of the image, I represents the pixel value of the input image, and the parameter γ can be learned continuously through backpropagation to adjust the brightness of the image. White balance is used to address the issue of different light reflections from the same object under different lighting conditions. It is achieved by multiplying the pixel values ​​of the three channels by a learned parameter (W). r W b W g The image's color proportions are adjusted using the following methods: α represents the contrast ratio between the brightest and darkest pixel values ​​in an image. Contrast is the difference between the brightest and darkest pixel values; a larger contrast indicates a higher overall image contrast, while a smaller contrast indicates that the pixels are closer together and the contrast is lower. P0 represents the image after contrast processing. Sharpening uses the Unsharp Mask (USM) algorithm, applying a Gaussian blur to the original image to remove minor interference details and noise. I represents the pixel value of the input image, and Gau(I(x)) represents the Gaussian blur processing.

[0052] Step 2: Using the enhanced image from Step 1 as the input vector and the defect location and category as the output vector, a deep neural network is used for deep learning to fit a deep model for defect detection. The output defect location and category are then used as the final result of the workpiece defect detection. An attention mechanism and feature fusion module are introduced into the deep neural network, allowing the network to focus on fusing effective features between channels during training. This integrates shallow features with deep semantic information, achieving accurate multi-scale defect detection. The deep neural network's workpiece defect detection consists of the following four main steps:

[0053] Step 2-1: The enhanced image from Step 1 is used as input for training and learning through a deep learning backbone network. The deep learning backbone network is used to extract image features. An object detection algorithm is selected as the main structure of the model, and a deep neural network is chosen as the backbone network. The feature pyramid, i.e., the multi-scale feature map part, constructs feature maps of different scales by passing the features extracted by the backbone network through a series of convolutional layers.

[0054] Step 2-2: Introduce an attention mechanism, namely the SE Block, into the deep neural network to enable the network to focus more on fusing effective features between channels during training. The attention mechanism is introduced after the activation function and before the concatenation process. The feature layer input to the attention structure first passes through a global average pooling layer to obtain global spatial information, then passes through two fully connected layers and two activation functions to compress and increase the dimensionality of the global information. Finally, the weights of each channel are obtained and multiplied with the corresponding feature maps to obtain the correlation between channels.

[0055] Steps 2-3: Add a feature fusion module to the deep neural network. This fuses shallow features with deep semantic information, allowing the shallow feature maps used for small-scale object detection and localization to simultaneously contain both detailed and deep semantic information. The feature fusion structure can be represented by the following formula:

[0056] T f =Concat(f{X p},f{Up(X q )})

[0057] f(x i ) = ReLU(L2(x i ))

[0058] Up(x i ) = ReLU(BN(f 1×1 (Bilinear(x i )))

[0059] Where, x i Let represent the i-th feature map in the feature map set to be fused, Bilinear represents bilinear interpolation, and BN is batch normalization. ReLU is the activation function, f 1×1 For convolutional layers with a kernel size of 1, for feature maps to be fused that differ in size from the target size, a 1×1 convolutional layer is first used to reduce the dimensionality, followed by bilinear interpolation to upsample the smaller, deeper feature maps. `f` represents the normalization of all feature maps to be fused to the same range using L2 regularization, facilitating subsequent concatenation. The concatenation method is a chain operation, where the unupsampled feature map X is concatenated. p and the interpolated feature map X q The layers are concatenated along the channel dimension, and then a 3×3 convolutional layer is used to eliminate the aliasing effect after concatenation. f This is the result of fusing shallow features with deep semantic information.

[0060] Steps 2-4: Defect detection is performed using different features extracted from the deep network and the feature pyramid after feature fusion. The feature pyramid structure is used to extract feature maps at different scales to improve the detection effect of multi-scale targets. Dense sampling is performed on different locations of each fused feature map, using different localization box scales and aspect ratios during sampling. Object classification and prior box regression are performed. The object classification result is output as the defect category, and the position of the prior box is output as the defect location. Since target defects are usually small in size, and the localization information of small targets usually comes from shallow features, the feature map set output by the backbone network ResNet50 is denoted as {C1, C2, C3, C4}. The shallow feature layer C2 with a small scale is selected as the initial feature map of the feature pyramid. The feature pyramid is constructed according to the following formula:

[0061] P i+1 =F(f 3×3 (F(f 1×1 (P i ))))

[0062] F(x i ) = ReLU(BN(x i ))

[0063] Where i∈{0,1,2,3,4}, P0=C2, f 3×3 and f 1×1 These are convolutional layers with kernel sizes of 3 and 1, respectively, f 3×3 The stride of the convolutional layer is 2. BN is the batch normalization process, F(x) i The activation is performed on the batch-normalized results. ReLU is the activation function, which can suppress the vanishing gradient phenomenon caused by excessive network depth. The final output feature map set is represented as {P0, P1, P2, P3, P4, P5}, with feature map sizes of {38, 19, 10, 5, 3, 1}.

[0064] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for defect detection of a workpiece based on a deep neural network, the method comprising: Comprising the following steps, Step 1, preprocessing the collected workpiece image to generate a workpiece image dataset, and image enhancement is performed on the training images in the workpiece image dataset; In step 1, an image enhancement module composed of a parameter prediction module and a differentiable image processing module is constructed, and the image enhancement module is used to perform image enhancement on the training images; The parameter prediction module includes five convolutional layers and two fully connected layers, each convolutional layer is connected with an activation layer, the input is a low-resolution image after downsampling, and the output is the hyperparameters required by the differentiable image processing module; The differentiable image processing module uses brightness, contrast and hue Gamma correction, contrast transformation and white balance transformation algorithms, and sharpening algorithms to enhance image edges and detail features to enhance the image; Step 2, introducing an attention mechanism and a feature fusion module into the deep neural network, using deep neural network for deep learning, fitting the defect detection depth model, and outputting the defect position and defect category as the final result of workpiece defect detection; Specifically, the following sub-steps are included: Step 2.1, input the image enhanced in step 1 into the deep neural network backbone network for training and learning, and the backbone network is used to extract image features; In step 2.1, the backbone network is ResNet50; Step 2.2, a series of convolutional layers are used to construct feature maps of different scales based on the image features extracted by the backbone network, and the multi-scale feature maps are denoted as a feature pyramid; In step 2.2, the feature map set output by the backbone network ResNet50 is denoted as {C1, C2, C3, C4}, and the small-scale shallow feature layer is selected as the initial feature map of the feature pyramid, and the feature pyramid is constructed according to the following formula: as the initial feature map of the feature pyramid, and the feature pyramid is constructed according to the following formula: wherein, , , and are convolution layers with kernel size 3, 1 respectively, the step of the convolution layer is 2, BN is batch normalization processing, is activation after batch normalization processing, ReLU is an activation function, and the final output feature map set is represented as with feature map sizes of {38, 19, 10, 5, 3, 1} respectively. Step 2.3, a feature fusion module is added to the deep neural network to fuse the features of the shallow layer and the semantic information of the deep layer, that is, to splice the image features extracted by the backbone network and the feature maps in the feature pyramid, so that the shallow feature maps used for small-scale target detection and positioning can contain both details and deep semantic information; Step 2.4, using the feature pyramid after feature fusion in the deep neural network for defect detection. 2.The workpiece defect detection method based on deep neural network according to claim 1, wherein: In step 2.3, the feature fusion module can be represented by the following formula: wherein, denotes the i-th feature map to be fused, i denotes the bilinear interpolation method, BN is the batch normalization processing, ReLU is the activation function, is a convolution layer with a kernel size of 1, and for the feature map to be fused with a target size different from that of the feature map, the feature map is first reduced in dimension by a 1x1 convolution layer, and then the deep feature map with a smaller size is up-sampled by using the bilinear interpolation method, denotes that the feature map to be fused is normalized to the same range by L2 regularization, which facilitates subsequent splicing, and the L2 regularization is connected with ReLU an activation function; the splicing mode selects a series splicing operation, and the feature map which has not been up-sampled and the feature map which has been interpolated are spliced in the channel dimension, and then a 3x3 convolution layer is used to eliminate the aliasing effect after splicing, which is the result of the fusion of the shallow feature and the deep semantic information.​ 3. The method of claim 2, wherein: In step 2.3, the attention mechanism, i.e., SE Block, is also introduced to make the network pay more attention to the effective features between the fusion channels in the training process, which is introduced in L2 regularization ReLU After the activation function and before the concatenation processing.

4. The method of claim 3, wherein: The processing process of the attention mechanism is as follows: The input feature layer first obtains global spatial information through a global average pooling layer, then compresses and upgrades the global information through two fully connected layers and two activation functions, finally obtains the weight of each channel and multiplies it with the corresponding feature map, thereby obtaining the correlation between channels, so that the network pays more attention to the effective features between channels during training.

5. The method of claim 1, wherein: In step 2.4, different locations in the fused feature pyramid are densely sampled, different bounding box scales and aspect ratios are used during sampling, object classification and prior box regression are performed, the object classification result is output as the defect category, and the prior box position is output as the defect position.

6. The method of claim 1, wherein: The sharpening algorithm uses the USM algorithm.

Images