Surface defect detection method and device
By using a pre-trained defect detection model in product surface defect detection, using dense connection module and YoloV3's multi-scale prediction module, the slow detection speed problem caused by complex backbone networks in the prior art is solved, and faster defect detection is achieved .
Patent Information
- Application Number
- CN202110292168.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-05-19
- Filing Date
- 2021-03-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-03-18
AI Technical Summary
In the prior art, the object detection algorithm based on convolutional neural network is complex in product surface defect detection, resulting in slow detection speed.
Using a pre-trained defect detection model, the feature extraction network of this model includes multiple dense connection modules and YoloV3's multi-scale prediction module. The dense connection module superimposes the features of the intermediate layer and the input layer to improve the utilization of shallow features of the image. Reduce the complexity of the backbone network.
The speed of product surface defect detection is improved, and faster defect detection is achieved by simplifying network structure and improving feature extraction efficiency.
Smart Images

Figure CN112950613B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to a surface defect detection method and device. Background Art
[0002] In product production, various types of defects will inevitably occur on the surface of products and their components, such as misalignment, scratches, contamination, etc. Therefore, product surface defect detection has become an indispensable part of the quality inspection process in the product production process.
[0003] In the field of surface defect detection, target detection algorithms based on convolutional neural networks emerge in an endless stream, but most algorithms inherit the backbone networks and ideas of natural scene target detection algorithms (such as Faster-Rcnn, Yolo series, SSD, etc.). Their complex backbone networks (such as VGG and Resnet) lead to high computational costs and slow detection of product surface defects. Summary of the invention
[0004] The embodiments of the present disclosure provide a surface inspection method and device, which can improve the speed of product surface defect inspection.
[0005] According to a first aspect of the present disclosure, a surface detection method is provided, comprising: acquiring an image of an object to be detected; inputting the image of the object to be detected into a pre-trained defect detection model to obtain the defect position of the object to be detected, wherein the network structure of the pre-trained defect detection model comprises a feature extraction network and a prediction network, the feature extraction network comprises a plurality of connected densely connected modules, the densely connected module comprises an input layer, a plurality of intermediate layers, and an output layer, the output layer of the densely connected module is the input layer of the next densely connected module, the last intermediate layer of the densely connected module is superimposed with the input layer of the densely connected module and the input layer of the forward densely connected module to obtain the output layer of the densely connected module, and the prediction network is a multi-scale prediction module of YoloV3.
[0006] According to a second aspect of the present disclosure, a surface detection device is provided, comprising: an acquisition module, configured to acquire an image of an object to be detected; a detection module, connected to the acquisition module, and configured to input the image of the object to be detected into a pre-trained defect detection model to obtain the defect position of the object to be detected, wherein the network structure of the pre-trained defect detection model comprises a feature extraction network and a prediction network, the feature extraction network comprises a plurality of connected densely connected modules, the densely connected module comprises an input layer, a plurality of intermediate layers, and an output layer, the output layer of the densely connected module is the input layer of the next densely connected module, the last intermediate layer of the densely connected module is superimposed with the input layer of the densely connected module and the input layer of the forward densely connected module to obtain the output layer of the densely connected module, and the prediction network is a multi-scale prediction module of YoloV3.
[0007] The embodiment of the present invention provides a surface defect detection method and device, which processes the image of the object to be detected by a pre-trained defect detection model to obtain the defect position of the object to be detected, thereby realizing defect detection of the object to be detected. Since the feature extraction network of the pre-trained defect detection model includes multiple dense connection modules, and the dense connection modules can improve the utilization rate of shallow features of the image and reduce the complexity of the backbone network of the detection model, the problem of slow detection speed of product surface defects in the prior art due to the complex structure of the backbone network of the target detection algorithm such as VGG, Resnet, etc. is solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:
[0009] Figure 1 A flow chart of a surface defect detection method provided according to an embodiment of the present invention is shown;
[0010] Figure 2 A schematic structural diagram of an embodiment of a dense connection module according to this embodiment is shown;
[0011] Figure 3 shows a backbone network including four densely connected modules according to this embodiment;
[0012] Figure 4 A comparison diagram of feature fusion of feature pyramid in prediction network of YoloV3 and feature fusion of feature pyramid in prediction network provided by this embodiment is shown;
[0013] Figure 5A flow chart of another surface defect detection method provided by an embodiment of the present invention is shown;
[0014] Figure 6 A schematic structural diagram of a surface defect detection device provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0015] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0016] It should be understood that the terms "first", "second", "third", "fourth", etc. in the claims, specifications and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.
[0017] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.
[0018] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0019] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0020] like Figure 1 As shown, an embodiment of the present invention provides a surface defect detection method, comprising:
[0021] Step 101: Acquire an image of an object to be detected.
[0022] In this embodiment, an image of the object to be detected can be obtained by taking photos or the like. For example, the object to be detected can be a lithium battery, steel, etc. Different objects to be detected have different corresponding surface defect types. For example, if the object to be detected is a lithium battery, the corresponding surface defect types include scratches, depressions, pits, etc.; if the object to be detected is steel, the corresponding surface defect types include cracks, inclusions, plaques, iron oxide indentations, scratches, etc.
[0023] Step 102: input the image of the object to be detected into a pre-trained defect detection model to obtain the defect position of the object to be detected.
[0024] The network structure of the pre-trained defect detection model includes a feature extraction network and a prediction network. The feature extraction network contains multiple connected densely connected modules, and the prediction network is a multi-scale prediction module of YoloV3.
[0025] Among them, the dense connection module includes an input layer, multiple intermediate layers, and an output layer. The output layer of the dense connection module is the input layer of the next dense connection module. The last intermediate layer of the dense connection module is superimposed with the input layer of the dense connection module and the input layer of the forward dense connection module to obtain the output layer of the dense connection module.
[0026] Figure 2 FIG. 4 shows a schematic diagram of a structure of an embodiment of a dense connection module according to this embodiment. Figure 2 As shown, each dense connection module includes an input layer, two intermediate layers (intermediate layer 10, intermediate layer 20), and an output layer. Among them, the output layer of dense connection module 1 is composed of the input layer of dense connection module 2, and the output layer of dense connection module 1 is composed of the intermediate layer 20 of dense connection module 1 and the input layer of dense connection module 1. Among them, the output layer of dense connection module 2 is composed of the input layer of dense connection module 2 (that is, the output layer of dense connection module 1), the input layer of dense connection module 1, and the intermediate layer 20 of dense connection module 2.
[0027] It should be noted that the present embodiment does not limit the number of densely connected modules. The input layer and output layer of each densely connected module are connected in a head-to-tail manner, and each densely connected module is sorted in the order of feature extraction. The forward densely connected module of a densely connected module refers to all densely connected modules arranged before the densely connected module. Figure 2Taking the dense connection module shown as an example, dense connection module 1 is the first module to perform feature extraction, so it is arranged at the front; dense connection module 2 is the second module to perform feature extraction, so the forward dense connection module of dense connection module 2 is dense connection module 1.
[0028] Preferably, the backbone network of the network structure of the pre-trained defect detection model includes 4 densely connected modules. Figure 3 FIG. 4 shows a backbone network including four densely connected modules according to this embodiment. Figure 3 As shown in the figure, the backbone network is divided into 5 stages, including the convolution layer and pooling layer in the first stage and the last 4 dense connection stages. Each dense connection stage includes a 1*1 convolution layer, a maximum pooling layer and a dense connection module. The 1*1 convolution layer reduces the number of feature channels to 64, which are input into the dense connection module through maximum pooling. The downsampling rates corresponding to each dense connection stage are 4, 8, 16 and 32 respectively. The backbone network needs more layers and larger receptive fields in the middle and late stages to extract effective features of medium and large targets. For this reason, the number of dense connection modules in the second to fifth stages gradually increases from 1 to 4.
[0029] Among them, YoloV3's prediction module uses a feature pyramid network to perform multi-scale feature object detection. When predicting objects on each scale feature map, YoloV3 uses k-means clustering to obtain several anchor boxes of different sizes, preset boxes for each grid unit on the output feature map, and judge positive and negative examples and border regression based on the intersection and union ratio of the preset box and the real box.
[0030] The initial model of the defect detection model in this embodiment can be trained by supervised learning, and the label is the surface defect type and position of the object to be tested. The initial model of the defect detection model can be an untrained defect detection model or an untrained defect detection model, and each layer of the initial model of the defect detection model can be set with initial parameters, and the parameters can be continuously adjusted during the training process. The feature extraction network of the initial model of the defect detection model can include multiple connected densely connected modules in this embodiment, and the prediction network of the initial model of the defect detection model is a multi-scale prediction module of YoloV3.
[0031] Among them, the image of the object to be detected is input into a pre-trained defect detection model to obtain the surface defect position of the object to be detected.
[0032] The embodiment of the present invention provides a surface defect detection method, which processes the image of the object to be detected by a pre-trained defect detection model to obtain the defect position of the object to be detected, thereby realizing defect detection of the object to be detected. Since the pre-trained defect detection model includes multiple dense connection modules, and the dense connection modules can improve the utilization rate of shallow features and reduce the complexity of the backbone network, the problem of slow detection speed of product surface defects in the prior art due to the complex structure of the backbone network of the existing target detection algorithm such as VGG, Resnet, etc. is solved.
[0033] In some optional implementations of this embodiment, the dense connection module includes an input layer, a first intermediate layer, a second intermediate layer, and an output layer. The operation between the input layer and the first intermediate layer includes convolution using a 1*1 convolution kernel, and the operation between the first intermediate layer and the second intermediate layer includes convolution using a 3*3 convolution kernel. Among them, each dense connection module includes four network layers. First, the number of channels of the input feature is reduced to 64 using a 1*1 convolution, and then new features are learned using a 3*3 convolution, and finally the features of this layer and the input features are superimposed. Using 1*1 convolution to reduce the network width can keep the network lightweight when the number of densely connected modules increases. The densely connected module in this embodiment reduces the density of connections and only superimposes the input features of each module on the output, thereby achieving the purpose of streamlining the structure and propagating features.
[0034] In some optional implementations of this embodiment, the operations between the input layer and the intermediate layer and / or between the intermediate layers of the densely connected module include convolution, batch normalization, and activation, wherein the activation function used for the activation operation is Leaky-Relu.
[0035] In some optional implementations of the present embodiment, the input feature map corresponding to each channel superposition operation in the prediction network of the pre-trained defect detection model also includes: a downsampled feature map having a resolution consistent with the upsampled feature map corresponding to the channel superposition operation, wherein the downsampled feature map is a feature map obtained by downsampling the feature map of the corresponding resolution in the feature extraction network.
[0036] Figure 4 The figure shows the comparison of the feature fusion of the feature pyramid in the prediction network of YoloV3 and the feature fusion of the feature pyramid in the prediction network provided by this embodiment. Among them, the prediction network of YoloV3 adopts the feature pyramid network structure, predicts the target at three scales of downsampling 8, 16 and 32, and the specific details of feature fusion are as follows Figure 4As shown in (a): First, 1*1 convolution is used to reduce the number of channels for the feature map with smaller resolution, and then upsampling is used to expand the feature map to the size of the target feature map, and finally the two feature maps are superimposed. This method only fuses the feature map with smaller resolution, which provides more semantic features, but the structural features are insufficient, which is not conducive to detecting small targets and accurate positioning. Therefore, this embodiment improves the feature pyramid structure of YoloV3 and adds feature fusion of feature maps with larger resolution, such as Figure 4 (b) As shown in the dotted box part, first, in order to retain the structural features of the feature map with a larger resolution, the feature map is downsampled by channel reorganization, and then the number of channels is reduced by 1*1 convolution, and finally this part is added to the original feature fusion module.
[0037] Continue to refer Figure 5 , which shows another surface defect detection method provided according to an embodiment of the present invention. Figure 5 As shown, the method includes:
[0038] Step 501: Acquire an image of the object to be detected. Figure 1 The steps 101 shown are similar and will not be described in detail here.
[0039] Step 502: determine an image of a region to be detected from the image of the object to be detected.
[0040] In this embodiment, the method of acquiring the image of the area to be detected can be a method based on multi-angle template matching on normalized cross-correlation lines, a method of feature point matching, or a method of contour matching, which will not be described in detail here.
[0041] The method comprises the following steps: step S101: performing pyramid down sampling on a pre-constructed template image containing an object to be detected and an image of the object to be detected; step S102: successively matching the top image of the image of the object to be detected with the top image of the template image with preset translation and rotation steps to obtain the position and rotation angle with the highest similarity; step S103: returning to the next layer of image, taking the position and rotation angle of the previous layer as a reference, successively matching in a neighborhood with preset translation and rotation degree steps to obtain the position and rotation angle with the highest similarity; and step S104: repeatedly executing step S103 to output the image of the area to be detected.
[0042] Among them, the template matching based on normalized cross-correlation uses the grayscale similarity between the template image and the image to be matched to find the best matching position, and its formula is shown as follows:
[0043]
[0044]
[0045]
[0046] Among them, s(x,y) represents the similarity between the template image t and the image to be matched f at point (x,y), and its value ranges from -1 to 1. The larger the value, the higher the similarity. t(i,j) represents the grayscale value of point (i,j) on the template image. represents the grayscale mean of the template image; f(x+i,y+j) represents the grayscale value of the point (x+i,y+j) of the image to be matched. Represents the grayscale mean of the region of interest in the image to be matched.
[0047] In this embodiment, the detection area is positioned by using online multi-angle template matching based on normalized cross-correlation, without the need to create a large number of templates offline, which greatly reduces the program's demand for memory.
[0048] Step 503: Input the image of the area to be inspected into the pre-trained defect detection model to obtain the defect position of the object to be inspected. Figure 1 The step 102 shown is similar and will not be described in detail here.
[0049] In some optional implementations of the present embodiment, the above step 102 includes: processing the image of the object to be detected by a pre-trained defect detection model to obtain at least one initial defect position and a corresponding confidence level; according to a preset confidence threshold, processing at least one initial defect position by non-maximum suppression to obtain the defect position of the object to be detected.
[0050] Among them, when using the pre-trained defect detection model to detect the image, for the prediction box with confidence higher than the threshold, non-maximum suppression is used to delete the redundant prediction objects. Non-maximum suppression is a local maximum search method. Non-maximum suppression in target detection is used to delete redundant prediction boxes, and its process is as follows: 1) Sort the prediction box set B of each type of target in descending order according to the confidence of the prediction box; 2) Select the prediction box M with the highest confidence in set B, delete it from set B and add it to set D; 3) Calculate the intersection and union ratio of the prediction box M and the prediction box of set B, and delete the prediction box whose intersection and union ratio is greater than the set threshold; 4) Repeat steps 2 and 3 until set B is an empty set.
[0051] Continue to refer Figure 6, which shows a surface defect detection device provided according to an embodiment of the present invention, including: an image acquisition module 601, used to acquire an image of an object to be detected; an image detection module 602, connected to the image acquisition module, used to input the image of the object to be detected into a pre-trained defect detection model to obtain the defect position of the object to be detected, wherein the network structure of the pre-trained defect detection model includes a feature extraction network and a prediction network, the feature extraction network includes a plurality of connected dense connection modules, the dense connection module includes an input layer, a plurality of intermediate layers, and an output layer, the output layer of the dense connection module is the input layer of the next dense connection module, the last intermediate layer of the dense connection module is superimposed with the input layer of the dense connection module and the input layer of the forward dense connection module to obtain the output layer of the dense connection module, and the prediction network is a multi-scale prediction module of YoloV3.
[0052] In this embodiment, the process of implementing surface defect detection through the image acquisition module 601 and the image detection module 602 is similar to steps 101 and 102 in a surface defect detection method provided in an embodiment of the present invention, and will not be described in detail here.
[0053] In some optional implementations of this embodiment, the input feature map corresponding to each channel superposition operation in the prediction network also includes: a downsampled feature map having a resolution consistent with the upsampled feature map corresponding to the channel superposition operation, wherein the downsampled feature map is a feature map obtained by downsampling the feature map of the corresponding resolution in the feature extraction network.
[0054] The embodiment of the present invention provides a surface defect detection device, which processes the image of the object to be detected through a pre-trained defect detection model to obtain the defect position of the object to be detected, thereby realizing defect detection of the object to be detected. Since the pre-trained defect detection model includes multiple dense connection modules, and the dense connection modules can improve the utilization rate of shallow features and reduce the complexity of the backbone network, the problem of slow detection speed of product surface defects in the prior art due to the complex structure of the backbone network of the existing target detection algorithm such as VGG, Resnet, etc. is solved.
Claims
1. A surface defect detection method, characterized in that: include: Acquire an image of the object to be detected; Inputting the image of the object to be detected into a pre-trained defect detection model to obtain the defect position of the object to be detected; The network structure of the pre-trained defect detection model includes a feature extraction network and a prediction network; The feature extraction network comprises a plurality of connected densely connected modules, wherein the densely connected modules comprise an input layer, a plurality of intermediate layers, and an output layer, wherein the output layer of the densely connected module is the input layer of the next densely connected module, and the output layer of the densely connected module is obtained by superimposing the last intermediate layer of the densely connected module, the input layer of the densely connected module, and the input layer of the forward densely connected module; The prediction network is a multi-scale prediction module of YoloV3; The input feature map corresponding to each channel superposition operation in the prediction network also includes: a downsampled feature map with the same resolution as the upsampled feature map corresponding to the channel superposition operation, wherein the downsampled feature map is a feature map obtained by downsampling the feature map of the corresponding resolution in the feature extraction network.
2. The method according to claim 1, characterized in that The densely connected module includes an input layer, a first intermediate layer, a second intermediate layer, and an output layer. The operation between the input layer and the first intermediate layer includes convolution using a 1*1 convolution kernel, and the operation between the first intermediate layer and the second intermediate layer includes convolution using a 3*3 convolution kernel.
3. The method according to claim 1, characterized in that: The operations between the input layer and the intermediate layer and / or between the intermediate layers of the densely connected module include convolution, batch normalization, and activation, wherein the activation function used for the activation operation is Leaky-Relu.
4. The method according to claim 1 or 2, after acquiring the image of the object to be detected, the method further comprises: Determine an image of a region to be detected from the image of the object to be detected; The step of inputting the image of the object to be detected into a pre-trained defect detection model to obtain the defect position of the object to be detected includes: The image of the area to be inspected is input into a pre-trained defect detection model to obtain the defect position of the object to be inspected.
5. The method according to claim 4, characterized in that Determining the image of the area to be detected from the image of the object to be detected comprises: Based on normalized cross-correlation template matching, the image of the area to be detected is located from the image of the object to be detected.
6. The method according to claim 5, characterized in that The normalized cross-correlation-based template matching to locate the image of the area to be detected from the image of the object to be detected includes: Step S101: performing pyramid down sampling on a pre-constructed template image containing an object to be detected and an image of the object to be detected; Step S102: sequentially matching the top image of the image of the object to be detected with the top image of the template image at preset translation and rotation steps to obtain the position and rotation angle with the highest similarity; Step S103: Return to the next layer of images, take the position and rotation angle of the previous layer as the reference, and successively match in the neighborhood with preset translation and rotation degree steps to obtain the position and rotation angle with the highest similarity; Step S104: Repeat step S103 to output the image of the area to be detected.
7. The method according to claim 1 or 2, characterized in that: The step of inputting the image of the object to be detected into a pre-trained defect detection model to obtain the defect position of the object to be detected includes: Processing the image of the object to be detected by using a pre-trained defect detection model to obtain at least one initial defect position and a corresponding confidence level; According to a preset confidence threshold, the at least one initial defect position is processed in a non-maximum suppression manner to obtain the defect position of the object to be detected.
8. A surface defect detection device, characterized in that: include: An acquisition module is configured to acquire an image of an object to be detected; A detection module, connected to the acquisition module, is configured to input the image of the object to be detected into a pre-trained defect detection model to obtain the defect position of the object to be detected, The network structure of the pre-trained defect detection model includes a feature extraction network and a prediction network. The feature extraction network comprises a plurality of connected densely connected modules, wherein the densely connected modules comprise an input layer, a plurality of intermediate layers, and an output layer, wherein the output layer of the densely connected module is the input layer of the next densely connected module, and the output layer of the densely connected module is obtained by superimposing the last intermediate layer of the densely connected module, the input layer of the densely connected module, and the input layer of the forward densely connected module. The prediction network is a multi-scale prediction module of YoloV3; The input feature map corresponding to each channel stacking operation in the prediction network also includes: A downsampled feature map having the same resolution as the upsampled feature map corresponding to the channel stacking operation, wherein the downsampled feature map is a feature map obtained by downsampling the feature map of the corresponding resolution in the feature extraction network.
Citation Information
Patent Citations
Workpiece apparent defect detection method based on machine vision
CN106204614A
A method for improving the detection accuracy of small targets based on characteristic spectral scale transformation
CN109308458A
A method and system for detect defects of mechanical part
CN109472790A
Method for generating object detection model
CN110084313A
Surface smudginess detection method, terminal device and storage medium
CN110245697A
Cited By
Battery shell quality judgment method based on battery steel before rolling release
CN119666930A
