A precise defect detection method and system based on point set representation

Abstract

The present invention relates to an accurate defect detection method and system based on point set representation. The method includes using point sets to represent the detection target; for the detection target represented by point sets, a global context feature pyramid network is used to enhance the extraction of multi-scale defect features with obvious contrast, a receptive field pyramid module is used to obtain and fuse defect features of different scales, and an adaptive positive and negative sample allocation detection head is used to screen defect positive samples for learning and training; based on the screened defect positive samples and the extracted defect features, accurate defect categories and precise defect locations are generated to realize the detection function. On this basis, a deep separable convolution is used instead of ordinary convolution to perform lightweight optimization on the network to improve the detection speed. The present invention solves the detection ambiguity problem caused by low contrast, solves the singleness problem caused by large defect size changes, and avoids the training problem caused by unbalanced positive and negative samples.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision image processing, and in particular relates to an accurate defect detection method and system based on point set representation. Background Art

[0002] Defect detection is a basic but extremely important task in the field of computer vision. Its purpose is to classify and locate the type and location of defects in images containing defects. The current mainstream defect detection methods can be mainly classified into target detection methods and image segmentation methods. In this invention, the focus is on target detection methods. With the development of target detection, it can be further divided into anchor-based and anchor-free methods according to whether the overall architecture uses a preset anchor frame.

[0003] At present, the mainstream defect detection methods are all based on anchor box detectors, and they are all detectors based on the classic target detection network architecture proposed for a specific defect. They are not very versatile and accurate. There are still many challenging problems in this field, such as Figure 1 As shown in the figure. First, defects in production and life are usually very similar to the background, which will cause low contrast, making it difficult for the detector to distinguish between the foreground and the background; second, the shapes and sizes of different defects or the same defects are often different, resulting in large variations in defect sizes, making it impossible for the same detector to accurately detect defects of all sizes; third, on a high-resolution image, the defect area often only occupies a small part, which will lead to an extremely unbalanced problem of positive and negative samples in the network training stage, making it impossible for the detector to model the characteristics of the defects well. Summary of the invention

[0004] Aiming at the problems of low contrast, large size variation and imbalance of positive and negative samples existing in defect detection methods in real scenes, the present invention proposes an accurate defect detection method and system based on point set representation.

[0005] The technical solution adopted by the present invention is as follows:

[0006] An accurate defect detection method based on point set representation comprises the following steps:

[0007] 1) The detection target is represented by a point set;

[0008] 2) For detection targets represented by point sets, the Global Context Feature Pyramid Network (GC-FPN) is used to enhance the extraction of multi-scale defect features with obvious contrast;

[0009] 3) For detection targets represented by point sets, the Receptive Field Pyramid Block (RFPB) is used to effectively acquire and fuse defect features of different scales;

[0010] 4) For detection targets represented by point sets, an Adaptive Positive and Negative Samples Allocation detection head is used to efficiently screen the very few positive samples (defects) used for learning and training;

[0011] 5) Based on the positive samples obtained in step 4) and the sample features extracted in steps 2) and 3), an accurate defect category and precise defect location are generated to realize the detection function;

[0012] Furthermore, the method also includes step 6): on the basis of realizing the detection function, for the feature extraction process of step 2), deep separable convolution is used to replace ordinary convolution to perform lightweight optimization on the network to achieve a faster detection purpose.

[0013] Furthermore, in step 2), the global context feature pyramid network, the network input is an image containing defects (i.e., the detection target), and the network can capture the long-distance constraint relationship between the defect and other positions during feature upsampling and feature downsampling through the self-attention mechanism, thereby enhancing the contrast of various defects, thereby solving the problem of being unable to distinguish between foreground and background due to low contrast in current defect detection.

[0014] Furthermore, the receptive field pyramid module in step 3) is composed of two branches, namely, a multi-receptive field convolution branch and a residual branch. The size of the receptive field is closely related to the convolution kernel. In order to accurately model defects of different sizes, convolution kernels with different expansion rates are used in the multi-receptive field convolution branch to effectively extract defect features of large, medium and small sizes. At the same time, in order to retain the fusion features generated in the previous stage, a residual connection is designed, and the different features of the two branches are effectively fused at the output end, thereby achieving the purpose of accurately detecting multi-scale defects, so as to solve the problem of large differences in shape and size between and within defect classes during the detection process.

[0015] Furthermore, in step 4), the adaptive positive and negative sample allocation detection head adopts an adaptive positive and negative sample allocation strategy. There are two types of sample allocation strategies involved in the detection head of the present invention, namely, the screening of the initial point set samples and the screening of the intermediate representation anchor box samples. The initial point set samples are screened using the existing method corresponding to the feature point mapping. The present invention improves the latter screening, i.e., the screening of the intermediate representation anchor box samples. From the statistical characteristics, it can be seen that the mean of the sample can be used to judge the rationality of the prediction, and the variance of the sample can be used to judge the volatility of the prediction. Based on this, the present invention improves the original anchor box allocation strategy into an adaptive sample allocation selection strategy, which makes full use of the statistical characteristics of the defects, and can effectively select a small number of positive samples as training to avoid problems caused by sample imbalance.

[0016] Furthermore, the lightweight network method described in step 6) adopts deep separable convolution to replace the ordinary convolution in the original process, which consists of two processes: channel-by-channel convolution and point-by-point convolution. Under the premise of ensuring the accuracy of model detection, it can reduce the overall parameter amount of the model and improve the detection speed of target defects.

[0017] An accurate defect detection system based on point set representation implemented by the method of the present invention comprises:

[0018] The global context feature pyramid network module is used to enhance the extraction of multi-scale defect features with obvious contrast for detection targets represented by point sets using the global context feature pyramid network;

[0019] The receptive field pyramid module is used to obtain and fuse defect features of different scales for detection targets represented by point sets;

[0020] An adaptive positive and negative sample allocation module is used to select defect positive samples for learning and training using an adaptive positive and negative sample allocation detection head for detection targets represented by point sets;

[0021] The defect detection module is used to generate accurate defect categories and precise defect locations based on the screened defect positive samples and extracted defect features to achieve the detection function;

[0022] The lightweight convolution module is used to replace the ordinary convolution used in the global context feature pyramid network module with depthwise separable convolution, reduce the parameters of the detection network, and improve the detection speed while ensuring the detection accuracy.

[0023] The present invention focuses on the main challenges in the current defect detection field. By utilizing the attention mechanism to capture the long-distance constraint relationship between the defect position and other positions, the characteristics of the defect are enhanced, and the detection ambiguity problem caused by the low contrast problem is solved. The features of defects of different sizes are extracted by carefully designing the receptive field, and the features of defects of various sizes are effectively integrated to solve the problem of uniformity caused by large changes in defect size. By introducing an adaptive positive and negative sample allocation strategy in the detection head, the statistical characteristics of the defect are fully utilized to avoid the training problem caused by unbalanced positive and negative samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 .Challenging problems in the field of defect detection; among them (a) is the low contrast problem, (b) is the large size variation problem, and (c) is the sample imbalance problem.

[0025] Figure 2 .Visualization results of the global context feature pyramid network; where (a) is the input image, (b) to (d) are the feature map heat map representations of the GC-FPN feature pyramid from bottom to top.

[0026] Figure 3 .Framework diagram of the receptive field pyramid module.

[0027] Figure 4 . Overall flow chart of the method of the present invention; wherein (a) represents the global context feature pyramid network, C1-C3 represent the bottom-up feature maps initially extracted and generated by the backbone network, and P1-P3 represent the top-down feature maps generated after fusion, (a) represents the feature map of the final output; (b) represents the pyramid feature map; (c) represents the receptive field pyramid module; (d) represents the adaptive positive and negative sample allocation detection head. DETAILED DESCRIPTION

[0028] In order to describe the method of the present invention more clearly and completely, the present invention will be further described below in conjunction with specific embodiments of the present invention and the accompanying drawings.

[0029] refer to Figure 4 , is a flow chart of the method of the present invention. In order to flexibly represent different defects of different sizes and shapes, the present method as a whole adopts a detection paradigm not based on anchor boxes, and the initial representation of the detection target is represented by a series of discrete points (point sets). The detection process mainly includes the following steps:

[0030] Step 1: Global Context Feature Pyramid Network

[0031] In order to capture the long-distance constraint relationship between defects and other positions, the contrast information of defects is enhanced through the self-attention mechanism. Different from the original feature pyramid network, the present invention adds a global context extraction module in the upsampling and downsampling processes of the features, so that the contrast information is fully enhanced. Since the attention heat maps corresponding to defects in different positions are the same, the context modeling process can be simplified as shown in the following formula, that is, the long-distance constraint relationship between defects and other positions is captured according to the following formula, which is essentially a self-attention mechanism. Where x represents the feature of the input global context module, H×W is the number of positions in the feature map, in is the index of the query position, j represents all possible positions, and W v and W k Represents the linear transformation matrix. Specifically, GC represents the output corresponding to the query position of the input feature map, x in represents the query position of the input feature map, x j represents other positions of the input feature map, x n Represents other positions of the input feature map, n represents the index of the feature map position, H represents the height of the feature map, and W represents the width of the feature map.

[0032]

[0033] The feature map output by the final global context feature pyramid network can be expressed as

[0034]

[0035] in, represents the representation of the feature map generated by the global context pyramid module for subsequent detection, P i Represents the feature map generated by the initial extraction, Conv(P i ) represents the convolution operation on the feature map. Indicates that it is used to generate The lower feature map of Indicate that Max pooling operation.

[0036] Figure 2 The heat map information of the defects that the network focuses on after training is shown, which illustrates the effectiveness of this step for defect contrast enhancement.

[0037] Step 2: Receptive Field Pyramid Module

[0038] In order to enable the network to perceive defects of different sizes, the receptive field pyramid module consists of a multi-receptive field convolution branch and a residual branch. Figure 3This is the framework diagram of the receptive field pyramid module. Specifically, the multi-receptive field convolution branch includes three 3×3 dilated convolutions (Conv) with expansion rates of 1, 3, and 5 respectively, in order to extract defects of different sizes. This branch can be expressed as the following formula:

[0039]

[0040]

[0041] Among them, the first formula is to illustrate the calculation relationship between the expansion rate and the receptive field. o represents the output feature map size, i represents the input feature map size, p represents the padding size, d represents the expansion rate size, k represents the convolution kernel size, s represents the convolution step size, and P in and Represent the input and output of the branch respectively, DConv i () represents a dilated convolution with a dilation rate of i, Conv represents a convolution operation, and Concat represents a concatenation operation.

[0042] At the same time, in order to retain the global features extracted by GC-FPN, the residual branch can be expressed as the formula:

[0043]

[0044] in, Represents the output of the receptive field pyramid module, RELU represents the rectified linear unit, and w represents the weight of the multi-receptive field convolution branch.

[0045] Step 3: Adaptive positive and negative sample allocation strategy

[0046] In order to reduce the training impact of unbalanced positive and negative samples, an adaptive sample allocation strategy is used to replace the selection strategy for anchor box samples in the original detection head. The main principle is to make full use of the statistical characteristics of the defect. First, the predicted anchor box is calculated based on the L2 distance, and then the mean and variance of the IoU between the predicted anchor box and the true label are used as the threshold for sample selection. This can automatically screen out suitable positive samples for training, reducing the impact of too many negative samples. In the detection method based on point set representation proposed in the present invention, the problem of extremely unbalanced positive and negative samples in the field of defect detection can be well solved by designing the sample selection strategy for the initial representation - the point set, and the intermediate representation - the anchor box. The adaptive positive and negative sample allocation detection head is as follows: Figure 4 As shown in Figure (d), deformable convolution refers to convolution with variable kernel shape, APNSA refers to adaptive positive and negative sample allocation strategy, and offset refers to the position deviation of the deformable convolution generated during the learning process that is the same as the input feature map size.

[0047] The sample selection process is mainly divided into the following three steps:

[0048] 1) Calculate the intersection over union (IoU) value between the predicted anchor box and the true label;

[0049] 2) Calculate the mean and variance of the IoU values ​​of all anchor boxes, and use the sum as the threshold for sample selection;

[0050] 3) Based on the IoU and the threshold, samples with IoU greater than the threshold are identified as positive samples, and the remaining candidate samples are negative samples.

[0051] Step 4: Defect Detection

[0052] Based on the defect positive samples generated in step 3 and combined with the fine-grained defect features learned in steps 1 and 2, the loss function is continuously optimized through iterative training, and accurate defect categories and precise defect locations are generated through the classification and regression branches.

[0053] Experimental data: This method has achieved excellent detection performance on three challenging defect datasets. The detection accuracy rate reached 94.4% on the mobile phone surface defect dataset containing defects such as defects, pits, stains, oil stains and scratches; the detection accuracy rate reached 99.4% on the surface defect dataset containing 10 complex texture backgrounds; and the detection accuracy rate reached 76.5% on the 6 typical hot-rolled steel strip surface defect datasets containing cracks, impurities, plaques, pitting, oxidation, scratches, etc., which are better than other current defect detection methods based on detection.

[0054] Step 5: Lightweight convolution

[0055] In order to further reduce the network parameters and improve the detection speed of the model, the depth-wise separable convolution is used to replace the ordinary convolution in the feature extraction stage (i.e., step 1). Among them, the depth-wise separable convolution consists of two processes: channel-by-channel convolution and point-by-point convolution. In the channel-by-channel convolution process, each channel of the original input image is convolved by only one convolution kernel, and the number of channels of the generated feature map is the same as the number of input channels; in the point-by-point convolution process, each convolution operation will perform weighted summation of the feature map generated by the channel-by-channel convolution in the depth direction to generate a new feature map. From the results, the feature dimension of the output of the depth-wise separable convolution is the same as that of the ordinary convolution, but from the perspective of the convolution parameter amount, the parameter amount of the depth-wise separable convolution is (convolution kernel W×convolution kernel H×input channel number) + (1×1×input channel number×output channel number), while the parameter amount of the ordinary convolution is (convolution kernel W×convolution kernel H×input channel number×output channel number), which is much larger than the parameter amount of the depth-wise separable convolution. Therefore, using depth-wise separable convolution to replace the original ordinary convolution can further improve the detection speed of defective targets.

[0056] Based on the same inventive concept, another embodiment of the present invention provides an accurate defect detection system based on point set representation implemented by the method of the present invention, which includes:

[0057] The global context feature pyramid network module is used to enhance the extraction of multi-scale defect features with obvious contrast for detection targets represented by point sets using the global context feature pyramid network;

[0058] The receptive field pyramid module is used to obtain and fuse defect features of different scales for detection targets represented by point sets;

[0059] An adaptive positive and negative sample allocation module is used to select defect positive samples for learning and training using an adaptive positive and negative sample allocation detection head for detection targets represented by point sets;

[0060] The defect detection module is used to generate accurate defect categories and precise defect locations based on the screened defect positive samples and extracted defect features to achieve the detection function;

[0061] The lightweight convolution module uses deep separable convolution to replace the ordinary convolution used in the global context feature pyramid network module for the implemented detection network, reducing the number of model parameters and improving the detection speed.

[0062] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0063] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0064] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. An accurate defect detection method based on point set representation, characterized in that: The following steps are involved: The detection target is represented by a point set; For detection targets represented by point sets, a global context feature pyramid network is used to enhance the extraction of multi-scale defect features with obvious contrast; For detection targets represented by point sets, the receptive field pyramid module is used to obtain and fuse defect features of different scales. For detection targets represented by point sets, an adaptive positive and negative sample allocation detection head is used to select defect positive samples for learning and training; Based on the screened defect positive samples and extracted defect features, the defect category and defect location are generated to realize the detection function; The input of the global context feature pyramid network is an image containing defects. The long-distance constraint relationship between defects and other positions is captured by the self-attention mechanism during feature upsampling and feature downsampling, thereby enhancing the contrast of various defects. The global context feature pyramid network captures the long-distance constraint relationship between defects and other positions according to the following formula: Among them, GC represents the output corresponding to the query position of the input feature map, x in Indicates the input feature map query position, x j 、x n Indicates other positions of the input feature map, in is the index of the query position, n is the index of the feature map position, W v and W k Represents the linear transformation matrix, H represents the height of the feature map, and W represents the width of the feature map; The receptive field pyramid module consists of two branches, namely a multi-receptive field convolution branch and a residual branch; the multi-receptive field convolution branch uses convolution kernels with different expansion rates to effectively extract defect features of three sizes: large, medium and small; At the output, different features of the two branches are fused; The adaptive positive and negative sample allocation detection head adopts an adaptive positive and negative sample allocation strategy, and the sample selection process includes the following steps: Calculate the intersection over union (IoU) value between the predicted anchor box and the true label; Calculate the mean and variance of the IoU values ​​of all anchor boxes and use the sum as the threshold for sample selection; According to the IoU threshold, samples with IoU greater than the threshold are identified as positive samples, and the remaining candidate samples are negative samples; The defect category and defect location are generated based on the defect positive samples obtained by screening and the extracted defect features, including: according to the defect positive samples, combined with the extracted fine-grained defect features, the loss function is continuously optimized through iterative training, and the defect category and defect location are generated through classification and regression branches.

2. The method according to claim 1, characterized in that In the process of feature extraction using a global context feature pyramid network, a depthwise separable convolution is used to replace a common convolution; the depthwise separable convolution consists of two processes: channel-by-channel convolution and point-by-point convolution.

3. The method according to claim 1, characterized in that The receptive field pyramid module is expressed as: Among them, o represents the output feature map size, i represents the input feature map size, p represents the padding size, d represents the dilation rate size, k represents the convolution kernel size, s represents the convolution step size, P in and Represent the input and output of the multi-receptive field convolution branch, DConv i () represents the dilated convolution with a dilation rate of i, Conv represents the convolution operation, and Concat represents the concatenation operation; Represents the output of the receptive field pyramid module, RELU represents the rectified linear unit, and w represents the weight of the multi-receptive field convolution branch.

4. An accurate defect detection system based on point set representation using the method according to any one of claims 1 to 3, characterized in that: include: The global context feature pyramid network module is used to enhance the extraction of multi-scale defect features with obvious contrast for detection targets represented by point sets using the global context feature pyramid network; The receptive field pyramid module is used to obtain and fuse defect features of different scales for detection targets represented by point sets; An adaptive positive and negative sample allocation module is used to select defect positive samples for learning and training using an adaptive positive and negative sample allocation detection head for detection targets represented by point sets; The defect detection module is used to generate defect categories and defect locations based on the screened defect positive samples and extracted defect features to realize the detection function; A model acceleration module that uses depthwise separable convolution to replace ordinary convolution in the feature extraction process of the global context feature pyramid network module.

5. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 3 is implemented.

Images