Defect detection method based on convolutional neural network
By designing a convolutional neural network with a simple structure and combining TensorRT C++ and Openvino C++ interfaces for rapid inference, the existing defect detection methods are solved and the training time is long, and efficient and convenient defect detection is achieved.
Patent Information
- Application Number
- CN202411526405.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-21
AI Technical Summary
The existing machine vision-based defect detection methods have problems such as complex structure, long training time, high hardware requirements, and inconvenient on-site deployment.
A defect detection method based on convolutional neural network is designed, including input layer, residual network, downsampling, upsampling, full connection layer, extrusion layer, extension layer and output layer, using TensorRT C++ and Openvino C++ interfaces for Quick reasoning to achieve defect detection.
The method has simple structure, fewer parameters, short training time, low hardware requirements, can converge quickly, is suitable for on-site deployment, and has high detection accuracy.
Smart Images
Figure CN119494822A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of machine vision defect detection, and in particular relates to a defect detection method based on convolutional neural network. Background Art
[0002] In the manufacturing industry, product defect detection plays a vital role. Product defects include scratches, flaws, pores, leaks, breaks, and cracks. In addition to affecting the appearance of the product, these defects may also have a negative impact on product quality and even cause serious manufacturing safety risks.
[0003] At present, defect detection based on machine vision is mainly divided into traditional defect detection and deep learning-based defect detection. Traditional defect detection mainly uses template matching and blob analysis to extract abnormal areas, which has high requirements for early lighting and poor robustness. Defect detection based on deep learning has the characteristics of high robustness and low requirements for lighting. However, based on common semantic segmentation networks, such as FCN and UNet, the network is relatively complex and takes a long time to train. It requires a higher-level industrial computer and is not convenient for on-site deployment.
[0004] Therefore, a convolutional neural network with a relatively simple structure and easy deployment is needed to solve the problem of defect detection of industrial products. Summary of the invention
[0005] The main technical problem solved by the present invention is to provide a defect detection method based on convolutional neural network, which has a relatively simple structure, fewer parameters, low hardware requirements for training, fast convergence, short training time, and uses TensorRTC++ interface and Openvino C++ interface to complete fast reasoning on GPU and CPU to detect the defect location and area information of the product.
[0006] In order to solve the above technical problems, a technical solution adopted by the present invention is: a defect detection method based on convolutional neural network, comprising the following steps:
[0007] S1: Collect product appearance defect images and some good product images as a data set, divide them into training set, validation set and test set according to the proportion, and manually mark the defect areas to generate annotation information;
[0008] S2: Design a convolutional neural network, and input the defect image and annotation information in step S1 into the convolutional neural network for training;
[0009] The convolutional neural network includes an input layer, a residual network, a downsampling layer, an upsampling layer, a fully connected layer, a squeeze layer, an expansion layer and an output layer;
[0010] S3: Export the convolutional neural network and its weights trained in step S2 into an onnx format model;
[0011] S4: Use TensorRT C++ and Openvino C++ interfaces to load the onnx model exported in step S3, collect new product images, convert them into corresponding formats, and input them into the image input interfaces of TensorRT and Openvino. Use the computer's GPU and CPU for inference respectively to obtain the feature map of the new product image.
[0012] S5: Analyze the feature map obtained in step S4 to determine whether the new product contains defects and the location and area of the defects.
[0013] Specifically, the labeling information of the manually labeled defective area in step S1 is composed of a label name and a label point, and a mask bitmap is finally generated and input into the neural network.
[0014] Specifically, in step S1, the product appearance defect image is obtained by photographing the actual defective product by an industrial camera, and the defective area is determined according to the product production standard.
[0015] Furthermore, the convolutional neural network in step S2 includes:
[0016] Input layer: used to input normalized image data;
[0017] Residual network: Use the first two layers of ResNet to extract image features;
[0018] Downsampling: Downsample the output of the residual network four times to obtain a four-level feature map;
[0019] Upsampling: Use bilinear method to upsample the feature maps of the four levels respectively;
[0020] Fully connected layer: Fusion residual network output and four times upsampled feature maps to obtain more features;
[0021] Squeeze layer and expansion layer: Use the fire module of squeezeNet to continue squeezing and expanding the fused feature map;
[0022] Output layer: Convolution and upsampling operations are performed on the squeezed and expanded feature image to output a feature image with the same dimension as the input image.
[0023] Furthermore, the convolutional neural network uses a cross entropy loss function to calculate the loss and perform a prediction evaluation on each pixel; a stochastic gradient descent optimizer is used to update and calculate the network parameters of the model to minimize the loss.
[0024] Furthermore, the cross entropy loss function calculation formula is:
[0025] L (y,y ' )＝ -Σ (y[ i ] *l og(y ' [ i ]))
[0026] Where y is the probability distribution of the true label, y' is the probability distribution predicted by the model, and i is the accumulated sequence number.
[0027] Furthermore, in step S3, the model and weights of the convolutional neural network with the smallest loss are exported into an onnx format model, which can be loaded by other interfaces, and the image can be input for reasoning to obtain the feature map corresponding to the input image, thereby achieving faster reasoning and cross-programming language reasoning.
[0028] Furthermore, in step S4, the onnx format model is loaded using the TensorRT C++ interface and the Openvino C++ interface, the image data is normalized, and input into the convolutional neural network to obtain the corresponding feature map.
[0029] Furthermore, step S5 analyzes the feature map to generate a mask bitmap, and analyzes the mask bitmap to determine whether the input image contains defects and the location and area of the defects.
[0030] The beneficial effects of the present invention include at least the following points:
[0031] The convolutional neural network of the present invention includes an input layer, a residual network, a downsampling layer, an upsampling layer, a fully connected layer, a squeezing layer, an expansion layer and an output layer, has a relatively simple structure, fewer parameters, lower hardware requirements for training, can converge quickly, and has a short training time;
[0032] The present invention has low requirements on the sample size of the data set, and only dozens of images are needed to train the model; after collecting and annotating image defects, the model can be quickly trained, and after exporting it as an onnx model, the TensorRT C++ interface and Openvino C++ interface can be used to load the model. After the onnx model, the image of the new product is input, and the computer's GPU and CPU are used for reasoning to obtain the defect location and area information of the new product, with high detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 is a flow chart of a defect detection method based on a convolutional neural network of the present invention;
[0034] Figure 2 It is a structural schematic diagram of the convolutional neural network model of the present invention;
[0035] Figure 3 It is a schematic diagram of the structure of upsampling of the convolutional neural network model of the present invention. DETAILED DESCRIPTION
[0036] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.
[0037] Embodiment: A defect detection method based on convolutional neural network comprises the following steps:
[0038] S1: Collect product appearance defect images and some good product images as a data set, divide them into training set, validation set and test set according to the proportion, and manually mark the defect areas to generate annotation information;
[0039] In this embodiment, the good product image is used when testing the network;
[0040] The labeling information of the manually labeled defect area in step S1 consists of a label name and a label point, and finally generates a mask bitmap which is input into the neural network;
[0041] In step S1, a product appearance defect image is obtained by photographing an actual defective product with an industrial camera, and the defect area is determined according to the product production standard.
[0042] In the specific implementation, the defective images of defective products and some good product images taken by industrial cameras are collected, and the defective images are divided into training set, verification set and test set in the ratio of 70%, 15% and 15%, and the good product images belong to the test set. The defective area of the defective image is annotated with the image annotation tool to generate the defective area information consisting of the label name and the annotation points;
[0043] S2: Design a convolutional neural network, and input the defect image and annotation information in step S1 into the convolutional neural network for training;
[0044] The convolutional neural network includes an input layer, a residual network, a downsampling layer, an upsampling layer, a fully connected layer, a squeeze layer, an expansion layer and an output layer;
[0045] Input layer: used to input normalized image data;
[0046] Residual network: Use the first two layers of ResNet to extract image features;
[0047] Downsampling: Downsample the output of the residual network four times to obtain a four-level feature map;
[0048] Upsampling: Use bilinear method to upsample the feature maps of the four levels respectively;
[0049] Fully connected layer: Fusion residual network output and four times upsampled feature maps to obtain more features;
[0050] Squeeze layer and expansion layer: Use the fire module of squeezeNet to continue squeezing and expanding the fused feature map;
[0051] Output layer: Convolution and upsampling operations are performed on the squeezed and expanded feature image to output a feature image with the same dimension as the input image.
[0052] In the specific implementation, the input layer: unifies the image size of the defect image into an image of 640*640 pixels, arranges it in the order of the three RGB channels, and normalizes the pixels into floating-point data between 0 and 1;
[0053] Residual network: Using the first two layers of ResNet, the processed image data is first input into a convolution layer with a convolution kernel size of 3*3 to convert the 3-channel data into 64-channel data, and then the data is input into a two-layer residual network to extract the image features;
[0054] Downsampling: The downsampling network has four layers, each of which consists of a maximum pooling layer and a convolutional layer. The output of the residual network is downsampled four times to obtain a four-level feature map, thereby further extracting image features;
[0055] Upsampling: Upsampling also has four layers, and the four-level feature maps of the above downsampling are upsampled separately using the bilinear method;
[0056] Fully connected layer: fuses the feature maps upsampled four times to obtain more features;
[0057] Squeeze layer and expansion layer: Use the fire module of squeezeNet to continue squeezing and expanding the fused feature map;
[0058] Output layer: Convolution and upsampling operations are performed on the squeezed and expanded feature image to output a feature image with the same dimension as the input image.
[0059] The convolutional neural network uses a cross entropy loss function to calculate the loss and perform prediction evaluation on each pixel; a stochastic gradient descent optimizer is used to update and calculate the network parameters of the model to minimize the loss.
[0060] In specific implementation, the annotation information is converted into a mask bitmap, and the mask bitmap is scaled to the same size of 640*640 pixels as the image. The cross entropy loss function is used to calculate the difference between the feature map output by the neural network and the mask bitmap, and a prediction evaluation is performed on each pixel.
[0061] The cross entropy loss function calculation formula is:
[0062] L( y,y ' ) ＝ -Σ ( y [i]*l og ( y ' [i]))
[0063] Where y is the probability distribution of the true label, y' is the probability distribution predicted by the model, and i is the accumulated sequence number.
[0064] S3: Export the convolutional neural network and its weights trained in step S2 into an onnx format model;
[0065] In step S3, the model and weights of the convolutional neural network with the smallest loss are exported into an onnx format model. The model can be loaded by other interfaces, and the image can be input for reasoning to obtain the feature map corresponding to the input image, thereby achieving faster reasoning and cross-programming language reasoning.
[0066] S4: Use TensorRT C++ and Openvino C++ interfaces to load the onnx model exported in step S3, collect new product images, convert them into corresponding formats, and input them into the image input interfaces of TensorRT and Openvino. Use the computer's GPU and CPU for inference respectively to obtain the feature map of the new product image.
[0067] In step S4, the onnx format model is loaded using the TensorRT C++ interface and the Openvino C++ interface, the image data is normalized, and input into the convolutional neural network to obtain the corresponding feature map.
[0068] In specific implementation, the model can be loaded using the model loading function interface in the TensorRT C++ interface provided by NVIDIA and the Openvino C++ interface provided by Microsoft, and the image can be loaded using OpenCV. The image is scaled to a size of 640*640 pixels and arranged in the order of the three RGB channels. The pixels are normalized into floating-point data between 0 and 1. After the processed image data is input into the data input function interface of the TensorRT C++ interface and the Openvino C++ interface, the inference interface of the TensorRT C++ interface and the Openvino C++ interface is called to obtain the feature image data. Different interfaces can be used to choose whether to use the GPU or the CPU for inference according to different computer configurations.
[0069] S5: Analyze the feature map obtained in step S4 to determine whether the new product contains defects and the location and area of the defects.
[0070] Step S5 analyzes the feature map to generate a mask bitmap, and analyzes the mask bitmap to determine whether the input image contains defects and information such as the location and area of the defects.
[0071] In the specific implementation, the feature image data is processed to obtain a mask bitmap of 640*640 pixels. The mask bitmap is scaled back to the original image size using OpenCV and threshold segmentation is performed to obtain the defect location and area information of the original image.
[0072] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A defect detection method based on convolutional neural network, characterized in that: The following steps are involved: S1: Collect product appearance defect images and some good product images as a data set, divide them into training set, validation set and test set according to the proportion, and manually mark the defect areas to generate annotation information; S2: Design a convolutional neural network, and input the defect image and annotation information in step S1 into the convolutional neural network for training; The convolutional neural network includes an input layer, a residual network, a downsampling layer, an upsampling layer, a fully connected layer, a squeeze layer, an expansion layer and an output layer; S3: Export the convolutional neural network and its weights trained in step S2 into an onnx format model; S4: Use TensorRT C++ and Openvino C++ interfaces to load the onnx model exported in step S3, collect new product images, convert them into corresponding formats, and input them into the image input interfaces of TensorRT and Openvino. Use the computer's GPU and CPU for inference respectively to obtain the feature map of the new product image. S5: Analyze the feature map obtained in step S4 to determine whether the new product contains defects and the location and area of the defects.
2. The defect detection method based on convolutional neural network according to claim 1, characterized in that: The labeling information of the manually labeled defect area in step S1 consists of a label name and a label point, and finally a mask bitmap is generated and input into the neural network.
3. The defect detection method based on convolutional neural network according to claim 1, characterized in that: In step S1, a product appearance defect image is obtained by photographing an actual defective product with an industrial camera, and the defect area is determined according to the product production standard.
4. The defect detection method based on convolutional neural network according to claim 1, characterized in that: The convolutional neural network in step S2 includes: Input layer: used to input normalized image data; Residual network: Use the first two layers of ResNet to extract image features; Downsampling: Downsample the output of the residual network four times to obtain a four-level feature map; Upsampling: Use bilinear method to upsample the feature maps of the four levels respectively; Fully connected layer: Fusion residual network output and four times upsampled feature maps to obtain more features; Squeeze layer and expansion layer: Use the fire module of squeezeNet to continue squeezing and expanding the fused feature map; Output layer: Convolution and upsampling operations are performed on the squeezed and expanded feature image to output a feature image with the same dimension as the input image.
5. The defect detection method based on convolutional neural network according to claim 1, characterized in that: The convolutional neural network uses a cross entropy loss function to calculate the loss and perform prediction evaluation on each pixel; a stochastic gradient descent optimizer is used to update and calculate the network parameters of the model to minimize the loss.
6. The defect detection method based on convolutional neural network according to claim 5, characterized in that: The cross entropy loss function calculation formula is: L( y,y ' ) ＝ -S ( y [i]*l og ( y ' [i])) Where y is the probability distribution of the true label, y' is the probability distribution predicted by the model, and i is the accumulated sequence number.
7. The defect detection method based on convolutional neural network according to claim 1, characterized in that: In step S3, the model and weights of the convolutional neural network with the smallest loss are exported into an onnx format model. The model can be loaded by other interfaces, and the image can be input for reasoning to obtain the feature map corresponding to the input image.
8. The defect detection method based on convolutional neural network according to claim 1, characterized in that: In step S4, the onnx format model is loaded using the TensorRT C++ interface and the Openvino C++ interface, the image data is normalized, and input into the convolutional neural network to obtain the corresponding feature map.
9. The defect detection method based on convolutional neural network according to claim 1, characterized in that: Step S5 analyzes the feature map to generate a mask bitmap, and analyzes the mask bitmap to determine whether the input image contains defects and information such as the location and area of the defects.