Defect feature extraction method based on SobelConv and CoordGate

Abstract

The invention discloses a SobelConv and CoordGate-based flaw feature extraction method, and the method comprises the following steps: obtaining an original feature map, and constructing a flaw feature extraction method of double-branch parallel and left and right branch information interaction; in a left branch of the defect feature extraction method, an improved deformable convolution DConv and a designed calculation space convolution CoordGate block are used for carrying out adaptive reinforcement learning on defect features of SobeConv; in a right branch of the flaw feature extraction method, a designed calculation space variation convolution CoordGate block stack is used for learning flaw local space features; fusing defect features through multiple times of double-branch information interaction and feature splicing, and then constructing a residual connection layer to fuse with originally input defect features; jointly constructing a new flaw detection model by using the flaw feature extraction method and the RT-DETR; and constructing a flaw image data set, training and testing the flaw image data set by using the new flaw detection model, and observing a flaw detection result. The objective of the invention is to reduce the model parameter quantity, improve the calculation efficiency, reduce the calculation resource consumption, and improve the recognition rate of the defect detection model under the complex background.

Description

Technical Field

[0001] The invention belongs to the field of image defect detection, and in particular relates to a defect feature extraction method based on SobelConv and CoordGate. Background Art

[0002] Flaw detection is an important problem in image processing and is widely used in the field of industrial inspection. By detecting flaws in images, automated quality control can be better achieved. The core of flaw detection lies in how to effectively extract features from images so that flaws can be accurately identified and classified.

[0003] At present, feature extraction methods are mainly divided into two categories: (1) statistical-based methods, and (2) methods combined with deep learning technology. In statistical-based methods, various edge detection operators can extract edge information well, and edge information can often highlight the outline of defects. However, the features extracted by these operators alone are still limited in complex images. In addition, feature extraction in defect detection also depends on the spatial information in the image, and the operator has limited ability to extract spatial information. Convolution-based methods can obtain relevant information more effectively.

[0004] The current research status at home and abroad shows that some scholars have proposed methods that combine operators and deep learning. These methods improve the effect of feature extraction by fusing edge information and deep features. However, most of these methods currently fail to fully utilize the features extracted by the two, resulting in insufficient expression of image feature information under complex backgrounds, which in turn affects the recognition rate of defect detection, and also brings about problems such as increased resource consumption and reduced computing efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a defect feature extraction method based on SobelConv and CoordGate, aiming to reduce the structural redundancy of the feature extraction method, improve the computational efficiency, reduce the consumption of computing resources, and improve the recognition rate of the defect detection model under complex backgrounds.

[0006] The present invention is implemented in this way: a defect feature extraction method based on SobelConv and CoordGate comprises the following steps:

[0007] Step 1: Obtain the original feature map to construct a defect feature extraction method with dual-branch parallelism and left-right branch information interaction;

[0008] Step 2: In the left branch of the defect feature extraction method, the improved deformable convolution DConv and the designed computational space convolution CoordGate block are used to adaptively reinforce the learning of defect features on SobelConv;

[0009] Step 3: In the right branch of the defect feature extraction method, the designed computational space variation convolution CoordGate block stack is used to learn the local spatial features of the defect;

[0010] Step 4: Fuse the defect features through multiple dual-branch information interactions and feature concatenation, and then construct a residual connection layer to fuse the defect features of the original input;

[0011] Step 5: Use the defect feature extraction method and RT-DETR to jointly construct a new defect detection model;

[0012] Step 6: Build a defect image dataset, and use the new defect detection model to train and test the defect image dataset, and observe the defect detection results.

[0013] Further, the step 1 is specifically:

[0014] The original feature map is the feature map output by the previous layer in the network model, which includes the obtained feature matrix. The defect feature extraction method includes the SobelConv branch after reinforcement learning, the designed computational space convolution CoordGate branch, the maxpool layer, the Concat layer, the residual connection layer, and the ordinary convolution layer.

[0015] Further, the step 2 is specifically:

[0016] In the left branch of the defect feature extraction method, on the one hand, an improved DConv is used to convert a modulation scalar with a weight range from 0 to 1 into an unrestricted dynamic weight similar to a convolution, and the weight is used to learn the position offset of the sampling point, and the learning is continuously iterated and mapped to the feature map of the SobelConv branch, which is used for the reinforcement learning of the Sobel operator of SobelConv on the defect edge information; on the other hand, an adaptive learning weight is designed to control the number of channels of the gated mask input into the SobelConv branch by the CoordGate block, so as to selectively amplify or attenuate the filter.

[0017] Further, the step 3 is specifically as follows:

[0018] The right branch of the defect feature extraction method includes a designed computational space variation convolution CoordGate block and a maxpool layer; wherein the designed CoordGate block includes two branches, and it is necessary to stack the CoordGate blocks twice and adjust the output channel during construction;

[0019] Step 3-1: The first branch of the designed CoordGate block adaptively learns the size of the convolution kernel scaling and stores it in the channel of the feature map;

[0020] Step 3-2: The second branch of the designed CoordGate block designs the position encoding of the static coordinates and passes it through the MLP network structure to obtain the gated unit, while saving the parameters of the gated map;

[0021] Step 3-3: Send the features obtained by multiplying the two branches into the maxpool layer for pooling to obtain the output features.

[0022] Further, the step 5 is specifically as follows:

[0023] The feature extraction method is used to fuse with the AIFI module in RT-DETR to perform feature interaction within the scale, retain spatial information and defect edge information, and the remaining modules adopt the modules of the native RT-DETR.

[0024] Further, the step 6 is specifically as follows:

[0025] Step 6-1, construct a defect image dataset according to the general defect detection process method;

[0026] Step 6-2: Set corresponding hyperparameters for the fused RT-DETR model, and perform training and testing;

[0027] Step 6-3: Observe the results of defect detection.

[0028] The beneficial effects of the present invention are as follows: (1) The present invention effectively combines the advantages of two defect feature extraction methods, can effectively enhance the perception of defect edges, and can also effectively retain spatial information. Compared with previous feature extraction methods, it can better locate and identify defects in complex scenes. (2) The feature extraction method involved in the present invention not only improves the accuracy of defect detection, but also reduces model parameters, improves calculation efficiency, and reduces computing resource consumption. (3) The feature extraction method involved in the present invention is highly scalable and can be well integrated into other detection methods, such as RT-DETR involved in the present invention. In addition, the feature extraction method can also be used to construct a new model together with other detection methods, such as FCOS, FasterRcnn, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the overall process provided by an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the structure of a feature extraction module provided in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the redesigned RT-DETR structure provided in an embodiment of the present invention;

[0032] Figure 4 A training result diagram of a training data set constructed according to an embodiment of the present invention;

[0033] Figure 5 A graph showing experimental results of defect detection on a wine bottle defect dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0035] In the embodiment, the above method is applied to a wine bottle defect dataset, which has 9 types of defects, namely, broken bottle cap, broken edge of bottle cap, deformed bottle cap, swirled bottle cap, skewed label, wrinkled label, bubble label and abnormal inkjet coding. The defects in this defect dataset have complex background and disorderly defect distribution. Using this dataset can effectively verify the improvement of accuracy and reduction of resource consumption brought by our proposed method.

[0036] Figure 1 The flow chart of a defect feature extraction method based on SobelConv and CoordGate of the present invention is shown. As shown in the figure, the present invention is implemented as follows: a defect feature extraction method based on SobelConv and CoordGate comprises the following steps:

[0037] S1. Obtain the original feature map to construct a defect feature extraction method with dual-branch parallelism and left-right branch information interaction;

[0038] S2. In the left branch of the defect feature extraction method, the improved deformable convolution DConv and the designed computational space convolution CoordGate block are used to adaptively reinforce the learning of defect features on SobelConv;

[0039] S3, the defect feature extraction method uses the designed computational space variation convolution CoordGate block stack in the right branch to learn the local spatial features of the defect;

[0040] S4, through multiple double-branch information interactions and feature concatenation, the defect features are fused, and then the residual connection layer is constructed to fuse the defect features of the original input;

[0041] S5. Use the defect feature extraction method and RT-DETR to jointly construct a new defect detection model;

[0042] S6. Construct a defect image dataset, and use the new defect detection model to train and test the defect image dataset, and observe the defect detection results.

[0043] Furthermore, the original feature map obtained in step S1 is used to construct a defect feature extraction method with dual-branch parallelism and left-right branch information interaction, and its specific content is:

[0044] The original feature map is the feature map output by the previous layer in the network model, which contains the obtained feature matrix. The defect feature extraction method includes the SobelConv branch after reinforcement learning, the designed computational space convolution CoordGate branch, the maxpool layer, the Concat layer, the residual connection layer, and the ordinary convolution layer. The overall structure of the feature extraction method is as follows: Figure 2 shown.

[0045] Further, the left branch of the defect feature extraction method in step S2 is specifically:

[0046] In the left branch of the defect feature extraction method, on the one hand, an improved DConv is used to convert a modulation scalar with a weight range from 0 to 1 into an unrestricted dynamic weight similar to a convolution, and the weight is used to learn the position offset of the sampling point, and the learning is continuously iterated and mapped to the feature map of the SobelConv branch, which is used for the reinforcement learning of the Sobel operator of SobelConv on the defect edge information; on the other hand, an adaptive learning weight is designed to control the number of channels of the gated mask input into the SobelConv branch by the CoordGate block, so as to selectively amplify or attenuate the filter.

[0047] Among them, when SobelConv is calculated, the horizontal convolution kernel of the Sobel operator is first And the convolution kernel in the vertical direction Convert to a pytorch tensor and expand it to the same shape as the number of input channels. At the same time, assign the predefined sobel operator's convolution kernel to the corresponding convolution layer weight, and use the weight to learn the position offset d of the sampling point xy =(d x ,d y ) and offset coordinates (x',y') = (x+d x ,y+d y ), then define a three-dimensional convolution layer, set the convolution kernel size to 3×3, and apply it to the filtering in the horizontal and vertical directions respectively. The calculation formula is: G x =S x*F, G y =S y *F;

[0048] Where * represents the convolution operation, G x and G y Respectively represent the gradient values ​​of the feature map in the horizontal and vertical directions, which are used to generate new feature maps;

[0049] The obtained horizontal and vertical outputs are added, and then channel dimension reduction is performed. At the same time, the number of channels of the gated mask input into the SobelConv branch by the designed adaptive learning weight control CoordGate block is multiplied to obtain the feature map processed by the SobelConv branch.

[0050] Further, the right branch of the defect feature extraction method in step S3 is specifically as follows:

[0051] The right branch of the defect feature extraction method includes a designed computational space variation convolution CoordGate block and a maxpool layer; wherein the designed CoordGate block includes two branches, and it is necessary to stack the CoordGate blocks twice and adjust the output channel during construction;

[0052] S31: The first branch of the designed CoordGate block adaptively learns the size of the convolution kernel scaling and stores it in the channel of the feature map;

[0053] S32: The second branch of the designed CoordGate block designs the position encoding of the static coordinates and passes it through the MLP network structure to obtain the gated unit, while saving the parameters of the gated map;

[0054] S33: The features obtained by multiplying the two branches are sent to the maxpool layer for pooling to obtain the output features.

[0055] Furthermore, the feature fusion method of step S4 is specifically as follows:

[0056] Through multiple double-branch information interactions in step S2 and the feature concatenation and fusion obtained in steps S2 and S3, the defect feature F2 is obtained, and then a residual connection layer is constructed to fuse the defect feature F1 of the original input to obtain F1+F2, and then the channel dimension is reduced through 1×1 volume;

[0057] Further, the feature extraction method used in step S5 is integrated with the AIFI module in RT-DETR, specifically:

[0058] The feature extraction method is used to fuse with the AIFI module in RT-DETR to perform feature interaction within the scale, retain spatial information and defect edge information, and the remaining modules adopt the modules of the original RT-DETR. The overall structure of the fused defect detection model is as follows: Figure 3 shown.

[0059] Furthermore, the step S6 constructs a defect image dataset, and uses a new defect detection model to train and test the defect image dataset, and observes the defect detection results, specifically:

[0060] S61, constructing a defect image data set according to a general defect detection process method;

[0061] S62. Set corresponding hyperparameters for the fused RT-DETR model, and perform training and testing;

[0062] S63. Observe the results of defect detection. After the training is completed, you can also view the specific values ​​of relevant evaluation indicators such as P, R, mAP50, mAP50-90 and the defect image recognition status, such as Figure 4 The PR graph generated by the model during training reflects the recognition rate of each defect type by the trained model. Figure 5 As shown, the model can be seen in its recognition of wine bottle defects.

[0063] Because the traditional feature extraction method and the feature extraction method based on deep learning are easy to cause structural redundancy under the general structural combination, and the ability to express image feature information under complex background is insufficient, it is easy to cause the subsequent problems of low recognition rate and high consumption of computing resources when detecting defects. The feature extraction method designed by the present invention can make good use of the edge information of the defect while retaining the spatial information. In addition, the defect detection method using RT-DETR as the benchmark and the designed feature extraction method jointly construct a new model, which can reduce the number of model parameters, improve computing efficiency, reduce computing resource consumption, and improve the recognition rate of the defect detection model under complex backgrounds.

[0064] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative labor are still within the scope of protection of the present invention.

Claims

1. A defect feature extraction method based on SobelConv and CoordGate, It is characterized in that include: Step 1: Obtain the original feature map to construct a defect feature extraction method with dual-branch parallelism and left-right branch information interaction; Step 2: In the left branch of the defect feature extraction method, the improved deformable convolution DConv and the designed computational space convolution CoordGate block are used to adaptively reinforce the learning of defect features on SobelConv; Step 3: In the right branch of the defect feature extraction method, the designed computational space variation convolution CoordGate block stack is used to learn the local spatial features of the defect; Step 4: Fuse the defect features through multiple dual-branch information interactions and feature concatenation, and then construct a residual connection layer to fuse the defect features of the original input; Step 5: Use the defect feature extraction method and RT-DETR to jointly construct a new defect detection model; Step 6: Build a defect image dataset, and use the new defect detection model to train and test the defect image dataset, and observe the defect detection results.

2. The defect feature extraction method based on SobelConv and CoordGate as claimed in claim 1, It is characterized in that The step 1 is specifically as follows: The original feature map is the feature map output by the previous layer in the network model, which includes the obtained feature matrix. The defect feature extraction method includes the SobelConv branch after reinforcement learning, the designed computational space convolution CoordGate branch, the maxpool layer, the Concat layer, the residual connection layer, and the ordinary convolution layer.

3. The defect feature extraction method based on SobelConv and CoordGate as claimed in claim 1, It is characterized in that The step 2 is specifically as follows: In the left branch of the defect feature extraction method, on the one hand, an improved DConv is used to convert a modulation scalar with a weight range from 0 to 1 into an unrestricted dynamic weight similar to a convolution, and the weight is used to learn the position offset of the sampling point, and the learning is continuously iterated and mapped to the feature map of the SobelConv branch, which is used for the reinforcement learning of the Sobel operator of SobelConv on the defect edge information; on the other hand, an adaptive learning weight is designed to control the number of channels of the gated mask input into the SobelConv branch by the CoordGate block, so as to selectively amplify or attenuate the filter.

4. The defect feature extraction method based on SobelConv and CoordGate as claimed in claim 1, It is characterized in that The step 3 is specifically as follows: The right branch of the defect feature extraction method includes a designed computational space variation convolution CoordGate block and a maxpool layer; wherein the designed CoordGate block includes two branches, and it is necessary to stack the CoordGate blocks twice and adjust the output channel during construction; Step 3-1: The first branch of the designed CoordGate block adaptively learns the size of the convolution kernel scaling and stores it in the channel of the feature map; Step 3-2: The second branch of the designed CoordGate block designs the position encoding of the static coordinates and passes it through the MLP network structure to obtain the gated unit, while saving the parameters of the gated map; Step 3-3: Send the features obtained by multiplying the two branches into the maxpool layer for pooling to obtain the output features.

5. The defect feature extraction method based on SobelConv and CoordGate as claimed in claim 1, It is characterized in that The step 5 is specifically as follows: The feature extraction method is used to fuse with the AIFI module in RT-DETR to perform feature interaction within the scale, retain spatial information and defect edge information, and the remaining modules adopt the modules of the native RT-DETR.

6. The defect feature extraction method based on SobelConv and CoordGate as claimed in claim 1, It is characterized in that The step 6 is specifically as follows: Step 6-1, construct a defect image dataset according to the general defect detection process method; Step 6-2: Set corresponding hyperparameters for the fused RT-DETR model, and perform training and testing; Step 6-3: Observe the results of defect detection.

Images