A small-scale defect detection model efficiency improvement method based on background undersampling

Abstract

The present application relates to a kind of small-scale defect detection model efficiency promotion method based on background undersampling, the method comprises the following steps: step 1: input definition process, step 2: feature is calculated using feature backbone network to input image;Step 3: for the output feature of backbone network, it is input two task networks: center point prediction and width-height prediction, to estimate target rectangular frame;The technical solution adopts the background undersampling method proposed in the present application, can remove more background points in the pooling process of feature learning, retain more defect target points, to improve the detection accuracy of model.

Description

TECHNICAL FIELD

[0001] The present application relates to a lifting method, in particular to a small-scale defect detection model efficiency improvement method based on background down-sampling, and belongs to the field of machine vision image processing. BACKGROUND

[0002] In recent years, the application of image detection technology based on deep learning in various fields has made great progress and achieved great success. Surface defect detection task is also an important application field of these technologies. At present, for the application of surface defect detection task, the general technology is to retrain the target detection technology on the surface defect data in the image, such as FasterRCNN, YOLO series method, CenterNet, ExtremeNet and method based on Transformer, etc. On this basis, strategies such as transfer learning, model lightening and knowledge distillation are used to improve the detection accuracy and speed of the model. In this series of methods, the calculation of image features is the basis of all methods and the key factor affecting the detection efficiency of the model. The optimization of feature learning network mainly includes the optimization of network layer connection mode, the optimization of convolution operation mode, and the optimization of feature map pooling mode. The target detection model optimization method based on background down-sampling proposed by the present application is also a way of pooling feature maps. At present, feature map pooling mainly has the following methods.

[0003] 1. Max-pooling. The feature map is divided into several matrix regions, and the maximum value of each region is taken as the result to form a new feature map input to the next layer network.

[0004] 2. Average-pooling. The feature map is divided into several matrix regions, and the average value of each region is taken as the result to form a new feature map input to the next layer network.

[0005] 3. Global average-pooling. A special form of average-pooling. The average value of the entire feature map is calculated as the result, and the average value of each channel takes a numerical value.

[0006] 4. Mixed-pooling. Average-pooling and max-pooling are mixed by random numbers.

[0007] 5. Stochastic-pooling. The feature map is converted into a probability map, and the features with high probability are selected as the result.

[0008] 6. DPP-pooling (Detail-Preserving Pooling). This pooling can amplify spatial changes and preserve important image structure details, and its internal parameters can be learned through backpropagation.

[0009] 7. Local importance pooling. The importance of each feature is calculated using an attention mechanism, which then discards unimportant features.

[0010] 8. Soft pooling. A softmax-based weighting method is used to preserve the basic properties of the input while amplifying larger intensity feature activations.

[0011] These methods are designed to balance the general image feature learning strategy, and the entire image is calculated. During the pooling process, no specific problem is analyzed and processed in a specific image area. In the present application, a background-based pooling strategy is proposed, which can effectively eliminate background features and enhance foreground features.

[0012] The existing problems in the prior art are as follows:

[0013] (1) In the surface defect detection scene of the equipment manufacturing field, the defects in the image are generally few, and the background is relatively single. In the feature learning network, the down-sampling operation is used for the defect area, which causes the adjacent defect targets to be discarded, and the recall rate of the model detection is reduced. The receptive field of the down-sampling operation in the background area is small; (2) The feature retention on the background area is too redundant, which leads to the decrease of the accuracy and efficiency of the detection model. (3) The existing target detection network structure is large in size, and the network with simple architecture has low detection precision. How to optimize the simple defect detection while ensuring the simplicity of the architecture and improving the efficiency. In view of the characteristics of the surface defect detection in the equipment manufacturing field, the present application proposes a target detection model efficiency optimization method based on background down-sampling, which solves the above problems and improves the model detection efficiency. SUMMARY

[0014] The present application is aimed at solving the problems in the prior art, and provides a small-scale defect detection model efficiency improvement method based on background down-sampling. The technical scheme combines the characteristics of the surface defect detection problem and the data itself, and proposes a target detection model efficiency optimization method based on background down-sampling, which is used to improve the efficiency of the existing small-scale neural network in surface defect detection.

[0015] In order to achieve the above purpose, the technical scheme of the present application is as follows: a small-scale defect detection model efficiency improvement method based on background down-sampling, which comprises the following steps:

[0016] Step 1: input definition process,

[0017] Step 2: calculating features based on the background down-sampling feature backbone network for the input image;

[0018] Step 3: input the output features of the backbone network into two task networks: center point prediction and width-height prediction, to estimate the target rectangular frame.

[0019] As an improvement of the present application, step 1: input definition process, as follows: for the input color image I ∈ R W×H×3 , where W × H represents the width and height of the image, the position of the defect target in the image is represented by a rectangular frame , where and respectively represent the top-left corner and bottom-right corner coordinates of the i-th defect target in the image, and in the calculation process of the model, the rectangular frame is converted into the center point and its distance to the rectangular frame , where represents the coordinates of the center point of the rectangular frame; respectively represent the distance from the center point to the rectangular frame, and the task of the detection model is to estimate the rectangular center point and its distance to the frame of each target in the image, thereby realizing the detection of the defect target.

[0020] As an improvement of the present application, step 2: calculating features using a feature backbone network for the input image, as follows: calculating features using a feature backbone network for the input image, in the feature backbone network, a convolutional neural network is used to realize it, which is stacked by convolutional blocks, where each convolutional block is composed of a convolutional layer and a pooling layer, and the convolutional layer uses 3*3 convolutional kernels of different depths to realize it.

[0021] As an improvement of the present application, the pooling layer is as follows:

[0022] First, for the input image, in the first convolutional block, two 3*3 convolutional layers with a depth of 128 are used, and after two convolutions, a feature map is obtained, where W1, H1, and d1 represent the width, height, and channel number of the feature map, respectively. For this feature map, it is input into the background decoder to calculate the background probability Y corresponding to each point in the feature map, and in the present application, the background decoder is composed of a 3*3 convolutional layer, a batch normalization layer, a Relu layer, a 1*1 convolutional layer, and a final Sigmod activation layer, as shown in Figure 2 .

[0023] Second, in order to realize the probability that each feature point in the feature map belongs to the background, in the training process, the rectangular frame of each defect target is mapped to the corresponding heat map representation through a Gaussian kernel , where the parameter 2 indicates that the heat map has two channels corresponding to the target class and the background, , where σ is the adaptive standard deviation of the target size, and similarly, , represents the value of each background target on the background heat map. For the loss of the decoder, the focal loss is used to train the model, as follows:

[0024]

[0025] where Y xyt represents the probability that the xy position prediction is of class t, is the label, in the present application, t only takes 0 or 1, and α and β are set to 2 and 4.

[0026] Finally, background down-sampling, in order to remove background points as much as possible when sampling, after obtaining the background probability, the background probability is fused with the features to form a new feature map, as follows, and maximum pooling with a step size of 2*2 is performed on the new feature map.

[0027] f i '(x, y) = f i (x, y) * Y i 0 (x, y)

[0028] where f i (x, y) represents the feature map output by the convolutional layer, Y i 0 (x, y) is a probability value representing that each point is a background point; f' i (x, y) is the feature map output by the background down-sampling module.

[0029] Convolutional block stacking: in the method of the present application, the input image is output with final features after passing through a network of 4 convolutional blocks, and the specific configuration of each convolutional block is as shown in Figure 4 For each convolutional block, background pooling needs to calculate background loss, therefore, the loss of the entire backbone network is defined as follows:

[0030]

[0031] As an improvement of the present application, step 3: the output features of the backbone network are input into two task networks: center point prediction and width-height prediction, to estimate the target rectangular frame, as follows:

[0032] For the center point prediction network and the width-height prediction network, the present application adopts a similar architecture to the background decoder, which is composed of a 3*3 convolutional layer, a batch normalization layer, a Relu layer, and two 1*1 convolutional layers, wherein the last two 1*1 convolutional layers, one is used to predict the width and height, and the other is used to predict the center point, for the convolutional layer used to predict the center point, a Sigmod layer is followed as the final output, as shown in Figure 5 Similarly, the center point coordinates need to be converted into the corresponding heat map wherein W4, H4, C represent the width, height of the feature map output by the fourth convolutional block and the number of categories of the defect target respectively, the focal loss and L1 loss are used to calculate the prediction loss of the center point and the width and height prediction loss respectively, and the specific calculation is as follows:

[0033]

[0034]

[0035] wherein Yc xyt represents the probability that the xy position prediction is the tth category of defect, represents the width and height of the rectangle predicted on the xy position, represents the label of the tth category of defect on the xy position, and N represents the total number of pixel points on the feature map. Finally, the overall loss of the method proposed in the application is defined as follows:

[0036] l=L bg +L c +L wh .

[0037] Compared with the prior art, the application has the following advantages: (1) the technical scheme adopts the background downsampling method proposed in the application, which can remove more background points and retain more defect target points in the pooling process of feature learning, thereby improving the detection accuracy of the model; (2) the technical scheme combines the background downsampling method, and the target detection model can achieve good detection accuracy on the basis of using a smaller scale and more concise feature learning network, thereby improving the inference efficiency of the model. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a schematic diagram of the network architecture of the application;

[0039] Figure 2 is a schematic diagram of the background decoder;

[0040] Figure 3 is a schematic diagram of the convolutional block;

[0041] Figure 4 is a schematic diagram of the backbone network;

[0042] Figure 5 is a schematic diagram of the center point and width and height estimation network. DETAILED DESCRIPTION

[0043] In order to deepen the understanding of the application, the following will describe the embodiment in detail in combination with the drawings.

[0044] Embodiment 1: see Figure 1A small defect detection model efficiency improvement method based on background down-sampling, the method comprises the following steps:

[0045] Step 1: input definition process,

[0046] Step 2: calculate the features of the input image using the feature backbone network based on background down-sampling;

[0047] Step 3: for the output features of the backbone network, input them into two task networks: center point prediction and width-height prediction, to estimate the target rectangular frame.

[0048] Wherein, step 1: input definition process, specifically as follows: for the input color image I∈R W×H×3 , wherein WxH represents the width and height of the image, and the position of the defect target in the image is represented by a rectangular frame , wherein and respectively represent the top-left corner and bottom-right corner coordinates of the i-th defect target in the image, and in the calculation process of the model, the rectangular frame is converted into the center point and its distance to the rectangular frame , wherein represents the coordinates of the center point of the rectangular frame; respectively represent the distance from the center point to the rectangular frame, and the task of the detection model is to estimate the rectangular center point and its distance to the frame of each target in the image, so as to realize the detection of the defect target.

[0049] Step 2: calculate the features of the input image using the feature backbone network, specifically as follows: calculate the features of the input image using the feature backbone network, in the feature backbone network, a convolutional neural network is used to realize it, which is stacked by convolutional blocks, wherein each convolutional block is composed of a convolutional layer and a pooling layer, and the convolutional layer uses 3*3 convolutional kernels with different depths to realize it.

[0050] Wherein, the pooling layer is specifically as follows:

[0051] First, for the input image, in the first convolutional block, two 3*3 convolutional layers with a depth of 128 are used, and after two convolutions, a feature map is obtained, wherein W1, H1 and d1 respectively represent the width, height and channel number of the feature map. For the feature map, input it into the background decoder to calculate the background probability Y corresponding to each point in the feature map, and the background decoder in the present application is composed of a 3*3 convolutional layer, a batch normalization layer, a Relu layer, a 1*1 convolutional layer and a final Sigmod activation layer, as shown in Figure 2 .

[0052] Second, in order to realize the probability of each feature point in the feature map belonging to the background, in the training process, the rectangular frame of each defect target is mapped to the corresponding heat map representation through the Gaussian kernel Wherein parameter 2 represents that the heat map has two channels corresponding to the target class and the background, The numerical value of each point on the heat map where the defect target is located, wherein sigma is the standard deviation of the target size self-adaptation, and similarly, The numerical value of each background target on the background heat map, for the loss of the decoder, the focal loss is used to train the model, as follows:

[0053]

[0054] Wherein Y xyt The probability of xy position prediction being the t-th class, Is the label, in the present application, t only takes 0 or 1 two classes, and alpha and beta are set to 2 and 4.

[0055] Finally, background undersampling, in order to remove background points as much as possible when sampling, after obtaining the background probability, the background probability is fused with the features to form a new feature map, as follows, and maximum pooling with a step of 2*2 is performed on the new feature map.

[0056] f′ i (x, y) = f i (x, y) * Y i 0 (x, y)

[0057] Wherein f i (x, y) represents the feature map output by the convolution layer, Y i 0 (x, y) is a probability value representing each point as a background point; f′ i (x, y) is the feature map output by the background undersampling module.

[0058] Convolution block stacking: in the method of the present application, the input image is output through a network of 4 convolution blocks, and the specific configuration of each convolution block is as shown in Figure 4 For each convolution block, the background loss needs to be calculated in the background pooling, therefore, the loss of the entire backbone network is defined as follows:

[0059]

[0060] As an improvement of the present application, step 3: the output feature of the backbone network is input into two task networks: center point prediction and width-height prediction, to estimate the target rectangular frame, as follows:

[0061] For the center point prediction network and the width and height prediction network, the application adopts a similar architecture as the background decoder, which is composed of a 3*3 convolution layer, a batch normalization layer, a Relu layer, two 1*1 convolution layers, wherein the last two 1*1 convolution layers, one is used for predicting the width and height, and the other is used for predicting the center point, for the convolution layer used for predicting the center point, a Sigmod layer is followed as the last output, as shown in Figure 5 It is also necessary to convert the center point coordinates into corresponding heat maps wherein W4, H4 and C respectively represent the width and height of the feature map output by the fourth convolution block and the number of categories of the defect target, for the center point prediction loss and the width and height prediction loss, the application respectively uses focal loss and L1 loss to calculate, as follows:

[0062]

[0063]

[0064] wherein Yc xyt represents the probability that the xy position prediction is the tth category of defect, represents the width and height of the rectangle predicted on the xy position, represents the label of the tth category of defect on the xy position, and N represents the total number of pixel points on the feature map. Finally, the overall loss of the method proposed by the application is defined as follows:

[0065] L = L bg + L c + L wh .

[0066] It should be noted that the above embodiments are not used to limit the protection scope of the application, and equivalent transformations or substitutions made on the basis of the above technical solutions all fall within the scope of protection of the claims of the application.

Claims

1. A method for improving efficiency of a small-scale defect detection model based on background undersampling, characterized in that, The method includes the following steps: Step 1: Enter the definition process, Step 2: Calculate features for the input image using a background-sampling-based feature backbone network; Step 3: Input the output features of the backbone network into two task networks: center point prediction and width and height prediction, to estimate the target rectangle; Step 1: input definition process, as follows: for input color image I ∈ R W×H×3 , where W x H represents the width and height of the image, the position of the defect target in the image is represented by a rectangular frame , where and represent the top-left and bottom-right point coordinates of the i-th defect target in the image, respectively, and in the calculation process of the model, the rectangular frame is converted into a center point and its distance to the rectangular frame , where represents the coordinates of the center point of the rectangular frame; , respectively, the task of the detection model is to estimate the center point of each target and its distance to the frame in the image, thereby realizing the detection of the defect target; Step 2: Calculate features for the input image using a feature backbone network, specifically as follows: Calculate features for the input image using a feature backbone network. In the feature backbone network, a convolutional neural network is used. This neural network is composed of stacked convolutional blocks, where each convolutional block consists of a convolutional layer and a pooling layer. The convolutional layers are implemented using 3*3 convolutional kernels of different depths. The pooling layer is as follows: First, for the input image, in the first convolutional block, two 3*3 convolution layers with a depth of 128 are used, and after two convolutions, a feature map is obtained where W1, H1, and d1 represent the width, height, and channel number of the feature map, respectively; for the feature map, it is input into the background decoder to calculate the background probability Y corresponding to each point in the feature map; Secondly, in order to realize the probability that each feature point in the feature map belongs to the background, in the training process, the rectangular frame of each defect target is mapped to the corresponding heat map representation through the Gaussian kernel Wherein parameter 2 represents that the heat map has two channels corresponding to the target class and the background respectively, The value of each point on the heat map where the defect target is located is represented by σ, which is the standard deviation of the target size, and similarly, The value of each background target on the background heat map is represented. For the loss of the decoder, the focal loss is used to train the model, as follows: where Y xyt represents the probability that the xy position prediction is of class t, is the label, t takes only the values 0 or 1, and a and b are set to 2 and 4, respectively. Finally, background sampling is performed. In order to remove background points during sampling, after obtaining the background probability, the background probability is fused with the feature to form a new feature map, as shown below. Then, max pooling with a stride of 2*2 is performed on the new feature map. f i (x, y) = f i (x, y) * Y i 0 (x, y) where f i (x, y) represents the feature map output by the convolutional layer, Y i 0 (x, y) is a probability value representing that each point is a background point; f′ i (x, y) is the feature map output by the background downsampling module; Convolutional Block Stacking: The input image is processed through a network of four convolutional blocks to output the final features. The first and second convolutional blocks have the same structure, each containing two 128*3*3 convolutional layers and one 2*2 background pooling layer. The third convolutional block consists of two 256*3*3 convolutional layers and one 2*2 background pooling layer. The last convolutional block consists of four 512*3*3 convolutional layers and one 2*2 background pooling layer. For each convolutional block, background loss needs to be calculated in the background pooling. Therefore, the loss of the entire backbone network is defined as follows: Step 3: Input the output features of the backbone network into two task networks: center point prediction and width / height prediction, to estimate the target bounding box, as follows: For the center point prediction network and the width and height prediction network, a similar architecture to the background decoder is adopted, consisting of a 3x3 convolutional layer, a batch normalization layer, a ReLU layer, and two 1x1 convolutional layers. The last two 1x1 convolutional layers are used to predict the width and height, and the other to predict the center point. The convolutional layer used for center point prediction is followed by a Sigmoid layer as the final output, which also needs to be used to obtain the center point coordinates. Convert to the corresponding heatmap In this paper, W4, H4, and C represent the width, height, and number of defect target categories of the feature map output by the fourth convolutional block, respectively. The prediction loss for the center point and the width / height prediction loss are calculated using focal loss and L1 loss, respectively, as follows: Among them, Yc xyt Let xy represent the probability that the predicted defect is of type t. This indicates that the label at position xy represents the type t defect. Let x and y represent the width and height of the predicted rectangle at position x and y, respectively, and N represent the total number of pixels in the feature map. Finally, the overall loss of the method proposed in this invention is defined as follows: L = L bg + L c + L wh .

Images