Adaptive box surface defect detection method based on deformable convolution

Abstract

The application relates to a deformable convolution-based adaptive frame surface defect detection method, in particular to the following steps: firstly, a surface defect detection module and a strip defect detection module based on deformable convolution are respectively constructed, and then the two modules are trained by using real surface defect images; in the detection stage, a surface defect image is subjected to the two kinds of defect detection modules respectively, and then the detection results are combined. Through the method, accurate detection of surface defects can be realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of surface defect detection, and particularly relates to a surface defect detection method based on deformable convolution adaptive box. BACKGROUND

[0002] Surface defects refer to various uneven and mutation phenomena on the surface of a material, including scratch, pit, bump and other defect types. Surface defects exist widely in industrial products and have adverse effects on the appearance, comfort and use performance of products, so surface defect detection has important economic significance.

[0003] Surface defects have various forms, random surface size, uneven shape, irregular distribution, and various types. At present, surface defects are mainly detected by manual visual inspection in factories. However, manual recognition of surface defects has serious problems such as shortage of personnel, low efficiency, and unavoidable subjective errors.

[0004] Currently, some factories also try to use traditional machine vision methods to detect surface defects, but traditional machine vision methods need to design detection features and judge data manually, so they are sensitive to changes in the detection environment. When the detection environment changes, the detection parameters need to be redesigned, which reduces the generalizability of traditional machine vision methods.

[0005] With the rapid development of deep learning technology, it has great advantages in object detection, and has been applied to face recognition, object detection in road scenes and other aspects. In these scenarios, the recognized objects often occupy the main position of the image and change little, so the deep learning algorithm can achieve good results for object detection. However, due to the random size, uneven shape, irregular distribution, and various types of surface defects, the existing algorithms cannot achieve ideal results when directly applied to surface defect detection.

[0006] In summary, the existing manual recognition of surface defects has the defects of low efficiency and poor objectivity, traditional machine vision methods cannot adapt to the diversification of the detection environment, and existing deep learning algorithms cannot be directly applied to surfaces with random size, uneven shape, irregular distribution, and various types to achieve ideal results. Therefore, there is an urgent need for a new surface defect detection method to overcome the defects of existing manual recognition of surface defects and traditional machine vision methods, and to combine with deep learning technology to obtain accurate and fast detection results according to the actual situation of the surface defects to be detected. SUMMARY

[0007] In order to solve the problems of low efficiency and poor objectivity of the manual surface defect identification method, the problem that the traditional machine vision method cannot adapt to the diversification of the detection environment, and the problem that the existing deep learning algorithm cannot be directly applied to the surface with random size, uneven shape, irregular distribution and various types, a surface defect detection method based on a deformable convolution adaptive frame is provided.

[0008] The technical scheme of the present application is as follows: a surface defect detection method based on a deformable convolution adaptive frame, comprising the following steps:

[0009] Step 1: constructing a deformable convolution unit DConv and a standard convolution unit SConv;

[0010] Step 2: based on the deformable convolution unit DConv and the standard convolution unit SConv constructed in step 1, a basic neural network model BNM is built;

[0011] Step 3: the basic neural network model BNM built in step 2 is trained by using a planar defect data set X1 in which planar defects are labeled, to obtain a planar defect detection module PDDM; the basic neural network model BNM built in step 2 is trained by using a strip defect data set X2 in which strip defects are labeled, to obtain a strip defect detection module SDDM;

[0012] Step 4: inputting a to-be-detected image into the trained PDDM model to obtain a planar defect detection result y e , and inputting the to-be-detected image into the trained SDDM model to obtain a strip defect detection result y r , combining the detection results y e and y r , and realizing the detection of surface defects.

[0013] Further, in step 1, the input of the deformable convolution unit DConv is a picture X D , and the output is a feature map Y D ; the processing process is as follows:

[0014] Step 1.1: the deformable convolution unit DConv performs an n x n convolution operation on the input picture, and the convolution has n 2 sampling positions in total, wherein for a position P0 in the picture X D , after the convolution operation, the offset ΔP n of the output sampling position is output, wherein the number of output channels is 2 x n 2 , and n 2 channels correspond to the horizontal direction offset of each sampling position, and the other n 2Each channel corresponds to the offset of the vertical direction of each sampling position; P0 is taken as the center position of the convolution sampling network, and the offset ΔP n , for each sampling position P n , the offset sampling position P' = P0 + P n + ΔP n is obtained, where P n is the sampling position relative to the center position determined according to the convolution operation;

[0015] Step 1.2: Obtain the feature value x(P') at the deformed irregular sampling position P';

[0016] Step 1.3: Use the convolution kernel w(P') to perform weighted summation on the feature values x(P') of all irregular sampling positions, and finally output the feature map Y D , wherein the position corresponding to P0 in the feature map Y D is y(P0) = ∑w(P')x(P').

[0017] Further, in step 1.2, if P' is a decimal position, then the response feature value is obtained by bilinear interpolation; the specific process is as follows:

[0018] For a decimal sampling point p, first perform one-dimensional interpolation in the horizontal and vertical directions respectively:

[0019] For each direction, find two integer positions near the sampling point, for the horizontal direction position P x of the sampling point, take and to calculate the interpolation weight g(q x0 , p x ) = 1 - |q x0 - p x |, g(q x1 , p x ) = 1 - |q x1 - p x |, then perform weighted summation on the sampling values x(q x0 ) and x(q x1 ) at the two integer positions to obtain the one-dimensional interpolation result G(p x , q x ) = g(q x0 , p x ) * x(q x0 ) + g(q x1 , p x ) * x(q x1 ) in the horizontal direction, and correspondingly obtain the one-dimensional interpolation result G(p y , q y ) in the vertical direction.

[0020] Then the one-dimensional interpolation results in horizontal and vertical directions are multiplied to obtain the bilinear interpolation characteristic value G(p, q) = G(p x ,q x ) * G(p y ,q y ) at the irregular sampling point.

[0021] Further, in step 1, the input of the standard convolution unit SConv is the picture X S , and the output is the feature map Y S ; the processing process is:

[0022] The standard convolution unit SConv performs two-dimensional convolution operation on the input picture, then performs batch normalization operation on the two-dimensional convolution operation output, and then obtains the output through the activation function.

[0023] Further, in step 2, the basic neural network model BNM is composed of standard convolution units SConv and deformable convolution units DConv connected alternately, and finally outputs the results through the full connection layer.

[0024] Further, in step 3, the defects that gather around the defect center and have a minimum circumscribed rectangle aspect ratio less than 2 are defined as planar defects; and the defects that have one end boundary far away from the defect geometric center and have a minimum circumscribed rectangle aspect ratio greater than 2 are defined as strip-shaped defects.

[0025] Further, in step 3, the process of obtaining the planar defect detection module PDDM and the strip-shaped defect detection module SDDM is:

[0026] (1) Planar defect detection module PDDM:

[0027] Step A: Construct the PDDM detection frame. For planar defects, the detection frame uses an elliptical frame. The parameters of the elliptical detection frame are (x e , y e , a e , b e , θ e ), where (x e , y e ) is the center point position of the detection frame, a e , b e are the long axis length and short axis length of the ellipse respectively, and θ e is the rotation angle of the elliptical frame.

[0028] Step B: Label the planar defects in the surface defect image using the elliptical frame. After labeling, the planar defect dataset X1 is constructed.

[0029] Step C: Train the base neural network model BNM using the dataset X1 marked with planar defects: Randomly initialize the base neural network model BNM parameters θ p0 , set the model training parameters: learning rate l1, decay rate λ1, total number of iterations T1, early stopping step s1, batch size m1; set the optimizer type and loss function L1 type;

[0030] Step D: Divide the planar defect dataset X1 into multiple small batch datasets B1 containing m1 training images, select a small batch dataset as the model input, where j represents the jth batch of data; calculate the metal surface defect feature detection model feature output and calculate the loss Then update the model parameters using the backpropagation algorithm;

[0031] Step E: Repeat step D, when the loss L1 does not decrease for s1 consecutive iterations, or when the iteration number t satisfies t > T1, stop the iteration, and obtain the trained planar defect detection module PDDM;

[0032] (2) Strip defect detection module SDDM:

[0033] Step a: Construct the SDDM detection frame, for strip defects, the detection frame uses a rectangular frame; determine the parameters of a rectangular detection frame as (x r ,y r ,w,h,θ r ), where (x r ,y r ) is the center point position of the detection frame, w, h are the width and height of the rectangular frame respectively, and θ r is the rotation angle of the rectangular frame;

[0034] Step b: Label the strip defects in the metal surface defect image using a rectangular frame, and construct the strip defect dataset X2 after labeling;

[0035] Step c: Train the base neural network model BNM using the dataset X2 marked with strip defects, randomly initialize the base neural network model BNM parameters θ r0 , set the training parameters: learning rate l2, decay rate λ2; total number of iterations T2, early stopping step s2, batch size m2; set the optimizer type and loss function L2;

[0036] Step d: Divide the training set X2 into multiple small batch datasets B2 containing m2 training images; select a small batch dataset as the model input, where j represents the jth batch of data, calculate the model output and calculate the loss Then the model parameters are updated using the back propagation algorithm.

[0037] Step e: repeat step d, when s2 consecutive iterations, loss L2 no longer decreases, or when the iteration number t satisfies t>T2, then stop iteration, get the trained strip defect detection module SDDM.

[0038] Inventive effects

[0039] The technical effect of the present application is that the present application proposes an adaptive frame surface defect detection method based on deformable convolution, which uses a deformable convolution DConv unit as the core of surface defect feature extraction to simulate the way of human brain processing visual information. In the surface defect detection method proposed in the present application, the surface defects are divided into planar defects and strip defects, and detection frames are designed for the two kinds of defects respectively, wherein the PDDM model is responsible for detecting planar defects, and the SDDM is responsible for detecting strip defects.

[0040] In order to simplify the network model design, the present application designs a basic neural network model BNM, and the PDDM model and the SDDM model are realized by changing the detection frame dimension of the BNM. Through the DConv and SConv units, the shallow contour information is extracted to the deep layer to form semantic information, realizing the interactive fusion of shallow information and deep information. At the same time, due to the use of attention mechanism, the extraction efficiency of surface defect features is improved, and the extraction ability of defect features is greatly strengthened.

[0041] The method proposed in the present application simulates the recognition mechanism of human brain from overall overview to local details for surface defects, improves the professional ability of network model for surface defect detection, increases the depth and diversity of the structure of the model, solves the difficulty of intelligent recognition of surface defects, and improves the accuracy and recall rate of surface defect detection.

[0042] Additional aspects and advantages of the present application will be given in the following description section, part of which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0043] Fig. 1 : Adaptive frame surface defect detection method based on deformable convolution implementation process

[0044] Fig. 2 : Basic neural network model structure

[0045] Fig. 3 : Surface defect detection experiment results using the present method DETAILED DESCRIPTION

[0046] The present application will be described below in conjunction with specific embodiments:

[0047] Referring to Figs. 1-3 The application provides a surface defect detection method based on a deformable convolution adaptive frame, which realizes surface defect identification and positioning by respectively constructing a surface defect detection network based on a deformable convolution and a strip defect detection module. The specific technical scheme is as follows:

[0048] Step 1: constructing a deformable convolution unit DConv and a standard convolution unit SConv; the input of the deformable convolution unit DConv is a picture X D , and the output is a feature map Y D ; the processing process is as follows:

[0049] Step 1.1: the deformable convolution unit DConv performs an n×n convolution operation on the input picture, and the convolution has n 2 sampling positions in total, wherein for a position P0 in the picture X D , after the convolution operation, the offset ΔP n of the output sampling position is obtained, wherein the number of output channels is 2×n 2 , n 2 channels correspond to the horizontal direction offset of each sampling position, and another n 2 channels correspond to the vertical direction offset of each sampling position; taking P0 as the center position of the convolution sampling network, and according to the offset ΔP n , for each sampling position P n , the offset irregular sampling position P' = P0 + P n + ΔP n is obtained, wherein P n is the sampling position relative to the center position determined according to the convolution operation;

[0050] Step 1.2: obtaining a feature value x(P') at the deformed irregular sampling position P'; if P' is a decimal position, a bilinear interpolation is performed to obtain the response feature value; the specific process is as follows:

[0051] For a decimal sampling point p, first, one-dimensional interpolation is performed in the horizontal and vertical directions respectively:

[0052] For each direction, two integer positions near the sampling point are found, and for the horizontal direction position P x of the sampling point, and and are taken to calculate the interpolation weight g(q x0 , p x ) = 1 - |q x0 - p x |, g(q x1 , p x ) = 1 - |q x1-p x |, then the weighted sum of the sample values x(q x0 ) and x(q x1 ) at the two integer positions is performed to obtain the one-dimensional interpolation result in the horizontal direction G(p x ,q x ) = g(q x0 ,p x ) x (q x0 ) + g(q x1 ,p x ) x (q x1 ), and the one-dimensional interpolation result in the vertical direction G(p y ,q y ) is also obtained accordingly.

[0053] Then, the one-dimensional interpolation results in the horizontal and vertical directions are multiplied to obtain the bilinear interpolation feature value G(p, q) = G(p x ,q x ) x G(p y ,q y ) at the irregular sampling point.

[0054] Step 1.3: The feature values x(P') at all irregular sampling positions are weighted and summed using the convolution kernel w(P'), and finally the feature map Y D is output, wherein the position corresponding to P0 in the feature map Y D is y(P0) = ∑w(P')x(P').

[0055] The input of the standard convolution unit SConv is a picture X S , and the output is a feature map Y S ; the processing process is as follows:

[0056] The standard convolution unit SConv performs a two-dimensional convolution operation on the input picture, then performs a batch normalization operation on the output result of the two-dimensional convolution operation, and finally obtains the output through an activation function.

[0057] Step 2: Based on the deformable convolution unit DConv and the standard convolution unit SConv constructed in step 1, a basic neural network model BNM is built; the basic neural network model BNM is composed of the standard convolution unit SConv and the deformable convolution unit DConv connected alternately, and finally the output result is output through a fully connected layer.

[0058] Step 3: Train the basic neural network model BNM built in step 2 using the planar defect dataset X1 with labeled planar defects to obtain a planar defect detection module PDDM; train the basic neural network model BNM built in step 2 using the strip defect dataset X2 with labeled strip defects to obtain a strip defect detection module SDDM; define a defect as a planar defect if it is gathered around the defect center and the aspect ratio of the minimum bounding rectangle is less than 2; define a defect as a strip defect if one end of the boundary is away from the geometric center of the defect and the aspect ratio of the minimum bounding rectangle is greater than 2.

[0059] The process of the planar defect detection module PDDM and the strip defect detection module SDDM is as follows:

[0060] (1) Planar defect detection module PDDM:

[0061] Step A: Construct a PDDM detection frame. For planar defects, use an elliptical frame. The parameters of the elliptical detection frame are (x e ,y e ,a e ,b e ,θ e ), where (x e ,y e ) is the center point position of the detection frame, a e , b e are the lengths of the major and minor axes of the ellipse, and θ e is the rotation angle of the elliptical frame.

[0062] Step B: Label the planar defects in the surface defect image using an elliptical frame. After labeling, construct a planar defect dataset X1.

[0063] Step C: Train the basic neural network model BNM using the dataset X1 with labeled planar defects: randomly initialize the parameters θ p0 of the basic neural network model BNM, set the model training parameters: learning rate l1, decay rate λ1, total number of iterations T1, early stopping step number s1, batch size m1; set the optimizer type and loss function L1 type;

[0064] Step D: Divide the planar defect dataset X1 into multiple small batch datasets B1 containing m1 training images, select a small batch dataset as the model input, where j represents the jth batch of data; calculate the metal surface defect feature detection model feature output and the loss Then update the model parameters using the backpropagation algorithm.

[0065] Step E: Repeat step D, when the loss L1 does not decrease for consecutive s1 iterations, or when the iteration number t satisfies t > T1, then stop iteration, and obtain the trained planar defect detection module PDDM;

[0066] (2) Strip defect detection module SDDM:

[0067] Step a: Construct the SDDM detection frame, and use a rectangular frame for strip defects; Determine the parameters of a rectangular detection frame as (x r ,y r ,w,h,θ r ), wherein (x r ,y r ) is the center point position of the detection frame, w and h are the width and height of the rectangular frame respectively, and θ r is the rotation angle of the rectangular frame;

[0068] Step b: Label the strip defects in the metal surface defect image using a rectangular frame, and construct a strip defect dataset X2 after labeling;

[0069] Step c: Train the basic neural network model BNM using the dataset X2 with labeled strip defects, randomly initialize the parameters θ r0 of the basic neural network model BNM, set the training parameters: learning rate l2, decay rate λ2; total iteration number T2, early stopping step s2, batch size m2; set the optimizer type and loss function L2;

[0070] Step d: Divide the training set X2 into multiple small batch datasets B2 containing m2 training images; Select a small batch dataset as the model input, wherein j represents the jth batch of data, calculate the model output , and calculate the loss Then update the model parameters using the back propagation algorithm;

[0071] Step e: Repeat step d, when the loss L2 does not decrease for consecutive s2 iterations, or when the iteration number t satisfies t > T2, then stop iteration, and obtain the trained strip defect detection module SDDM.

[0072] Step 4: Input the to-be-detected image into the trained PDDM model to obtain the planar defect detection result y e ; and input the to-be-detected image into the trained SDDM model to obtain the strip defect detection result y r , combine the detection results y e and y r , and realize the detection of surface defects.

[0073] Embodiments of the present application are described in detail below, which are exemplary and intended to explain the present application, and cannot be understood as a limitation of the present application.

[0074] The database used in the present embodiment is the industrial product surface images collected in the factory, and the total number is 200. The resolution of each image is 640*640.

[0075] Step 1: Construct the deformable convolution unit DConv and the standard convolution unit SConv.

[0076] Wherein, the input of the deformable convolution unit DConv is the picture X D , and the output is the feature map Y D ; the processing process is:

[0077] Step 1.1: The deformable convolution unit DConv performs a 3*3 convolution operation on the input picture, and the convolution has a total of 9 sampling positions, wherein for a position P0 in the picture X D , after the convolution operation, the offset ΔP n of the output sampling position is obtained, wherein the number of output channels is 18, 9 channels correspond to the horizontal direction offset of each sampling position, and the other 9 channels correspond to the vertical direction offset of each sampling position; take P0 as the center position of the convolution sampling network, and according to the offset ΔP n , for each sampling position P n , the offset irregular sampling position P' = P0 + P n + ΔP n is obtained, wherein P n is the sampling position relative to the center position determined according to the convolution operation.

[0078] Step 1.2: Obtain the feature value x(P') at the deformed irregular sampling position P'; if P' is a decimal position, then the corresponding feature value is obtained by bilinear interpolation; the specific process is as follows:

[0079] For the decimal sampling point p, first perform one-dimensional interpolation in the horizontal and vertical directions respectively:

[0080] For each direction, find two integer positions near the sampling point; for the horizontal direction position P x of the sampling point, take and to calculate the interpolation weight g(q x0 , p x ) = 1- |q x0 -p x |, g(q x1 , p x ) = 1- |q x1 -px |, then the weighted sum of the sample values x(q x0 ) and x(q x1 ) at these two integer positions is taken to obtain the one-dimensional interpolation result in the horizontal direction G(p x , q x ) = g(q x0 , p x ) x x(q x0 ) + g(q x1 , p x ) x x(q x1 ). y For the vertical direction position P and , the interpolation weight g(q y0 , p y ) = 1 - |q y0 - p y | and g(q y1 , p y ) = 1 - |q y1 - p y | are calculated, and then the weighted sum of the sample values x(q y0 ) and x(q y1 ) at these two integer positions is taken to obtain the one-dimensional interpolation result in the vertical direction G(p y , q y ) = g(q y0 , p y ) x x(q y0 ) + g(q y1 , p y ) x x(q y1 ).

[0081] The one-dimensional interpolation results in the horizontal and vertical directions are then multiplied to obtain the bilinear interpolation feature value G(p, q) = G(p x , q x ) x G(p y , q y ) at the irregular sampling point.

[0082] Step 1.3: The feature values x(P') at all irregular sampling positions are weighted and summed using the convolution kernel w(P'), and finally the feature map Y D is output, where the position corresponding to P0 in the feature map Y D is y(P0) = ∑w(P')x(P').

[0083] The input of the standard convolution unit SConv is the picture X S , and the output is the feature map Y SThe processing procedure is that the standard convolution unit SConv performs two-dimensional convolution operation on the input picture, the input channel number is c1, the output channel number is c2, the convolution kernel size is k, then the batch normalization operation is performed on the two-dimensional convolution operation output result, and the SiLU activation function is used to obtain the output.

[0084] Step 2: Based on the deformable convolution unit DConv and the standard convolution unit SConv constructed in step 1, a basic neural network model BNM is built. The basic neural network model BNM is a 9-layer neural network. Among them, the 1st, 3rd, 5th, and 7th layers are SConv units, the 2nd, 4th, 6th, and 8th layers are DConv units, and the 9th layer is a full connection layer (Full connected layer, Fc).

[0085] Step 3: The basic neural network model BNM built in step 2 is trained using the planar defect dataset X1 labeled with planar defects to obtain a planar defect detection module PDDM; the basic neural network model BNM built in step 2 is trained using the strip defect dataset X2 labeled with strip defects to obtain a strip defect detection module SDDM.

[0086] First, we define the defect as a planar defect if it is gathered around the defect center and the aspect ratio of the minimum bounding rectangle is less than 2; and define the defect as a strip defect if one end of the boundary is away from the geometric center of the defect and the aspect ratio of the minimum bounding rectangle is greater than 2.

[0087] The process of obtaining the planar defect detection module PDDM and the strip defect detection module SDDM is:

[0088] (1) Planar defect detection module PDDM:

[0089] Step A: Build a PDDM detection frame. For planar defects, the detection frame uses an elliptical frame. The parameters of the elliptical detection frame are (x e ,y e ,a e ,b e ,θ e ), where (x e ,y e ) is the center point position of the detection frame, a e , b e are the lengths of the major and minor axes of the ellipse, and θ e is the rotation angle of the elliptical frame.

[0090] Step B: Label the planar defects in the surface defect image using an elliptical frame. During the labeling process, the ellipse completely covers the planar defect as much as possible, and the elliptical frame is minimized. After labeling, the planar defect dataset X1 is constructed.

[0091] Step C: Train the base neural network model BNM using the dataset X1 marked with planar defects: Randomly initialize the parameters θ of the base neural network model BNM p0 , set the model training parameters: learning rate 0.01, decay rate 0.1; use the Adam optimizer, and select the cross-entropy loss function as the loss function; set the training parameters: total number of iterations 300, early stopping step number 50, and batch size 8.

[0092] Step D: Divide the planar defect dataset X1 into 35 small batch datasets B1 each containing 8 training images, and select one small batch dataset as the model input, where j represents the jth batch of data; calculate the feature output of the metal surface defect feature detection model and the loss Then update the model parameters using the backpropagation algorithm;

[0093] Step E: Repeat Step D, when the loss L1 no longer decreases for 50 consecutive iterations, or when the iteration number t satisfies t>300, stop the iteration, and obtain the trained planar defect detection module PDDM;

[0094] (2) Strip defect detection module SDDM:

[0095] Step a: Construct the SDDM detection frame, and use a rectangular frame for strip defects; determine the parameters of a rectangular detection frame as (x r ,y r ,w,h,θ r ), where (x r ,y r ) is the center point position of the detection frame, w and h are the width and height of the rectangular frame, respectively, and θ r is the rotation angle of the rectangular frame;

[0096] Step b: Label the strip defects in the metal surface defect image using a rectangular frame, and during the labeling process, the rectangular frame should completely cover the strip defects as much as possible, and the rectangular frame should be minimized. After labeling, construct the strip defect dataset X2;

[0097] Step c: Train the base neural network model BNM using the dataset X2 marked with strip defects, and randomly initialize the parameters θ of the base neural network model BNM r0 , set the training parameters: learning rate 0.01, decay rate 0.5, set the training parameters: total number of iterations 300, early stopping step number 50, and batch size 8; use the Adam optimizer as the optimizer, and use the cross-entropy loss function as the loss function.

[0098] Step d: Divide the training set X2 into 35 small batch datasets B2 each containing 8 training images; select one small batch dataset As model input, where j represents the jth batch of data, the model output is calculated And the loss is calculated Then the model parameters are updated using the backpropagation algorithm.

[0099] Step e: repeat step d, when the loss L2 no longer decreases for 50 consecutive iterations, or when the iteration number t satisfies t>300, then stop iteration, and obtain the trained strip defect detection module SDDM.

[0100] Step 4: when performing surface defect detection, input the image to be detected into the trained PDDM model to obtain the surface defect detection result y e ; and input the image to be detected into the trained SDDM model to obtain the strip defect detection result y r , combine the detection results y e and y r , and obtain the result of metal surface defect detection.

[0101] The performance of the surface defect test algorithm of 100 industrial product images actually taken from the factory is obtained. Through experimental verification, using the method, the inference time of a single image is 30ms, the detected surface defects account for 92% of the overall surface defect area, and the non-defect part accounts for 5 of the framed area. The experimental results show that the method proposed in the present application has high defect detection rate and low false detection rate, and the method performance is superior.

[0102] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the principles and purposes of the present application within the scope of the present application.

Claims

1. A deformable convolution based adaptive bounding box surface defect detection method, characterized in that, Comprising the following steps: Step 1: constructing a deformable convolution unit DConv and a standard convolution unit SConv; Step 2: based on the deformable convolution unit DConv and the standard convolution unit SConv constructed in step 1, building a basic neural network model BNM; wherein the basic neural network model BNM is composed of standard convolution units SConv and deformable convolution units DConv connected alternately, and finally outputs the results through a fully connected layer; Step 3: define the defects around the defect center as surface defects if the aspect ratio of the minimum bounding rectangle is less than 2; define the defects as strip defects if the aspect ratio of the minimum bounding rectangle is greater than 2, with one end boundary away from the geometric center of the defect; Using the surface defect dataset X1 marked with surface defects to train the basic neural network model BNM built in step 2 to obtain a surface defect detection module PDDM; using the strip defect dataset X2 marked with strip defects to train the basic neural network model BNM built in step 2 to obtain a strip defect detection module SDDM; The process of obtaining the surface defect detection module PDDM and the strip defect detection module SDDM is: (1) Surface defect detection module PDDM: Step A: Constructing PDDM detection frame, using ellipse frame for planar defects, the parameters of the ellipse detection frame are determined as wherein is the center point position of the detection frame, , are the long axis length and the short axis length of the ellipse respectively, is the rotation angle of the ellipse frame; Step B: Label the planar defects in the surface defect image using an elliptical frame. After labeling, construct a planar defect dataset ; Step C: Using the dataset labeled with the planar defects Training the base neural network model BNM: Randomly initializing the base neural network model BNM parameters Setting the model training parameters: learning rate , decay rate , total number of iterations , early stopping steps , batch size ; setting the optimizer type and loss function type; Step D: Splitting the planar defect data set into multiple mini-batch data sets containing one training image , selecting a mini-batch data set as model input, where denotes the th batch of data ; Computing metal surface defect feature detection model feature output and computing a loss then updating the model parameters using a backpropagation algorithm; Step E: repeat Step D when the loss does not decrease successively or when the number of iterations satisfies the iteration is stopped and the trained planar defect detection module PDDM is obtained. (2) Strip defect detection module SDDM: Step a: constructing SDDM detection frame, using rectangular frame for strip defects; the parameters of a rectangular detection frame are wherein is the center point position of the detection frame, , are the width and height of the rectangular frame, respectively, is the rotation angle of the rectangular frame; Step b: Label the strip defects in the metal surface defect image using a rectangular box, and construct a strip defect dataset after labeling ; Step c: using the dataset labeled with the bar defects Training the base neural network model BNM, randomly initializing the base neural network model BNM parameters Setting training parameters: learning rate , decay rate ; total number of iterations , early stopping steps , batch size ; setting the optimizer type and loss function ; Step d: split the training set into multiple mini-batch data sets containing a number of training images ; select one mini-batch data set as model input, where denotes the i-th data batch, compute the model output , and compute the loss , then update the model parameters using the backpropagation algorithm ​​ Step e: repeat step d, when the loss of the successive iterations does not decrease anymore, or when the number of iterations satisfies the iteration is stopped and the trained strip defect detection module SDDM is obtained.​ Step 4: input the image to be detected into the trained PDDM model to obtain a planar defect detection result y e ; and input the image to be detected into the trained SDDM model to obtain a strip defect detection result y r ; and combine the detection results y e and y r to realize detection of surface defects.

2. The adaptive frame surface defect detection method based on deformable convolution according to claim 1, wherein the specific process is: In step 1, the input of the deformable convolution unit DConv is a picture X D , and the output is a feature map Y D ; the processing process is: Step 1.1: The deformable convolutional unit DConv performs a process on the input image. Convolution operation, this convolution has a total of There are sampling locations, where for image X... D At a given position P0 in the image, after convolution, the output sampling position offset is... The number of output channels is , Each channel corresponds to the horizontal offset of each sampling position, and additionally... Each channel corresponds to a vertical offset at each sampling position; P0 is taken as the center position of the convolutional sampling network, and the offset ΔP is used as the basis for the offset. n For each sampling location P n The irregular sampling positions after offset are obtained. ,in To determine the sampling position relative to the center position based on the convolution operation; Step 1.2: Deformed irregular sampling locations Upper acquisition feature values ; Step 1.3: Utilizing convolution kernel Feature values for all irregular sampling positions Weighted sum is performed, and finally the feature map Y is output D Where the position corresponding to P0 in the feature map Y D is​ 3. The method of claim 2, wherein the deformable convolution is applied to the bounding box to generate a bounding box surface defect detection result. In step 1.2, if If the decimal position is a fraction, then bilinear interpolation is performed to obtain the characteristic value of the response. The standard convolution unit SConv performs a two-dimensional convolution operation on the input picture, then performs a batch normalization operation on the two-dimensional convolution operation output result, and then obtains the output through an activation function. For , first one-dimensional interpolation is performed in both horizontal and vertical directions respectively: For each direction, find two integer positions near the sample point, for the sample point horizontal direction position , take and , calculate the interpolation weight , , then weighted sum the sample values and on the two integer positions, get the one-dimensional interpolation result in the horizontal direction , and also get the one-dimensional interpolation result in the vertical direction ; The one-dimensional interpolation results in horizontal and vertical directions are then multiplied to obtain the bilinear interpolation characteristic value on the .

4. The method of claim 1, wherein the deformable convolution is applied to the adaptive bounding box. In step 1, the input of the standard convolution unit SConv is a picture X S , and the output is a feature map Y S ; the processing process is: ​

Images