Double-branch contour enhanced industrial product defect detection method based on small sample

Abstract

The invention discloses a double-branch contour enhanced industrial product defect detection method based on small samples. The method comprises the following steps: 1, constructing a positive sample image data set and a defect sample image data set of an industrial product; 2, constructing a background removing module, and removing backgrounds of the reference image and the image to be registered; 3, constructing an image rotation coarse registration module, and rotating the first to-be-registered image until the first to-be-registered image is aligned with the first reference image; 4, constructing a double-branch feature-level fine registration module, and solving a mean value and a variance of the feature map c as template features of the industrial product; 5, solving a feature map d and an abnormal score matrix of the same industrial product defect sample image; and 6, establishing an adaptive defect threshold segmentation network to obtain a defect segmentation image. According to the method, better balance between the detection precision and the model universality in a small sample scene is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of machine vision and relates to a double-branch contour enhancement type industrial product defect detection method based on small samples. Background Art

[0002] In the field of intelligent manufacturing and industrial automation, defect detection is a key link to ensure product quality. Machine vision technology based on deep learning is the mainstream key technology in the current defect detection field. However, in actual industrial scenarios, it faces many challenges: 1. Scarcity of defect samples: Due to strict quality control, there are fewer defective products, resulting in the number of defect samples available for training being far lower than normal samples. This imbalance makes it difficult to train a model that can accurately identify a few defects; 2. Diversity and unpredictability of defects: Industrial products may have a variety of defect types, ranging from tiny scratches to major structural damage. In addition, new or improved products may also bring new types of defects, making it almost impossible to collect a comprehensive dataset covering all potential defect types; 3. Improve the generalization ability of the model. The existing fully supervised model needs to have enough labeled samples for each defect type for training. For new sample types that are not observed during the training process, the performance of the model is significantly reduced when it is directly tested on them, and it cannot adapt to the modern industrial needs of rapid production and frequent product updates. Summary of the invention

[0003] The purpose of the present invention is to provide a dual-branch contour enhancement industrial product defect detection method based on small samples, which solves the problems of low model detection accuracy and weak generalization ability caused by small number of samples and difficult feature alignment in the prior art.

[0004] The technical solution adopted by the present invention is a double-branch contour enhancement industrial product defect detection method based on a small sample, comprising the following steps:

[0005] S1: Construct positive sample image dataset and defect sample image dataset of industrial products;

[0006] S2: Construct a background removal module, input two different positive sample images of the same industrial product, one as the reference image and the other as the image to be registered, use the Otsu binarization method and the Canny edge detection operator to remove the background of the reference image and the image to be registered respectively, and obtain the first reference image and the first image to be registered;

[0007] S3: constructing an image rotation coarse registration module, rotating the first image to be registered to be aligned with the first reference image, and obtaining a second image to be registered;

[0008] S4: Construct a dual-branch feature-level precise registration module, and after training with a positive sample image dataset, input the first reference image and the second image to be registered for precise alignment, obtain the feature map a of the first reference image and the feature map b of the second image to be registered, upsample the feature map a and the feature map b and perform channel dimension splicing to obtain the feature map c, solve the mean and variance of the feature map c, and use it as the template feature of the industrial product;

[0009] S5: Solve the feature map d of the defect sample image of the same industrial product, calculate the Mahalanobis distance between the feature map d and the template feature, and use it as the abnormal score matrix of the defect sample image;

[0010] S6: Establish an adaptive defect threshold segmentation network, solve the optimal global segmentation threshold, use the global segmentation threshold to segment the anomaly score matrix, and obtain a defect segmentation image.

[0011] The present invention is also characterized in that:

[0012] A dual-branch contour enhancement method for industrial product defect detection based on small samples. In S2, the Otsu binarization method and the Canny edge detection operator are used to remove the background of the reference image and the image to be registered, respectively. Specifically:

[0013] S2.1: Extract the target objects in the reference image and the image to be registered using the Otsu binarization method;

[0014] S2.2: Use the Canny edge detection operator to extract the edge information of the target object, and use the edge information to constrain the target object: starting from the surrounding edge pixels of the reference image and the image to be registered, traverse each pixel inward, and before encountering an edge pixel, set the traversed pixels to the background.

[0015] In the dual-branch contour enhancement method for industrial product defect detection based on small samples, the first image to be registered is rotated to be aligned with the first reference image in S3, specifically: the central axis angle α of the circumscribed rectangle of the maximum connected area of ​​the first reference image is calculated, and the central axis angle β of the circumscribed rectangle of the maximum connected area of ​​the first image to be registered is calculated. By comparing the angle difference between the central axis angle α and the central axis angle β, a rotation matrix is ​​constructed to rotate the first image to be registered to be aligned with the first reference image.

[0016] A dual-branch contour enhancement method for industrial product defect detection based on small samples. The dual-branch feature-level precise registration module in S4 is specifically to build a dual-subnet structure based on the ResNet-18 network. The two subnets share parameters. Each subnet adds a dual-branch contour enhancement spatial transformation network submodule after the first convolution block, and adds a spatial transformation network STN submodule after the second and third convolution blocks. The loss function uses a negative cosine loss function.

[0017] In S4, the mean and variance of the feature map c are solved using a multivariate Gaussian distribution estimator.

[0018] A dual-branch contour enhancement method for industrial product defect detection based on small samples, a dual-branch contour enhancement spatial transformation network submodule, consists of two branches: the first branch is the main branch, and the input is the original image fused with contour information, which is composed of two convolutions with a kernel size of 3×3, and each convolution layer is followed by a maximum pooling layer; the second branch is the auxiliary branch, and the input is an image containing only contour information extracted from the original image using the Canny edge detection operator, which is composed of convolutions with a kernel size of 3×3, and each convolution layer is followed by a maximum pooling layer; the main branch and the auxiliary branch process the two inputs respectively, extract spatial features through convolutional neural networks, and then fuse these spatial features through a fully connected layer to predict transformation parameters and construct a transformation matrix to align the image.

[0019] A dual-branch contour enhancement method for industrial product defect detection based on small samples is used. In S5, the feature map d of the defect sample image of the same industrial product is solved. Specifically, the reference image and a defect sample image of the same industrial product are processed by a background removal module, an image rotation coarse registration module and a dual-branch feature-level fine registration module respectively, and then the feature map d of the defect sample image is obtained after upsampling and channel dimension splicing.

[0020] A dual-branch contour enhancement method for industrial product defect detection based on small samples is proposed. The optimal global segmentation threshold is solved in S6. Specifically, the anomaly score matrix and the corresponding defect true mask map are input into the adaptive defect threshold segmentation network, and the defect threshold is automatically determined by the neural network. The defect threshold is subtracted from the anomaly score matrix, and soft binarization is performed using the activation function ReLU and normalization. The MSE loss between the predicted soft binary defect segmentation map and the defect true mask map is minimized to obtain the optimal global segmentation threshold for the anomaly score matrix.

[0021] The beneficial effects of the present invention are:

[0022] 1. Through background removal based on edge information constraints and image-level coarse alignment, the interference of sample background area changes on feature extraction is avoided and the difficulty of feature matching is reduced;

[0023] 2. The constructed dual-branch contour enhancement spatial transformation network submodule enhances the model's spatial feature perception ability and effectively reduces the detection error caused by incomplete alignment between images;

[0024] 3. The established adaptive defect threshold segmentation network reduces the dependence on labeled defect samples and improves the adaptability to new samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of the defect detection method of the present invention;

[0026] Figure 2 is a defect sample image of a tablet in Example 1 of the present invention;

[0027] Figure 3 is a reference image of the tablet in Example 1 of the present invention;

[0028] Figure 4 is a defect segmentation image of the tablet in Example 1 of the present invention;

[0029] Figure 5 is a defect sample image of a cable product in Embodiment 2 of the present invention;

[0030] Figure 6 is a reference image of the cable product in Embodiment 2 of the present invention;

[0031] Figure 7 is a defect segmentation image of a cable product in Embodiment 2 of the present invention;

[0032] Figure 8 is a defect sample image of a metal nut in Example 3 of the present invention;

[0033] Fig. 9 is a reference image of the metal nut in Example 3 of the present invention;

[0034] Fig.10 It is a defect segmentation image of the metal nut in Example 3 of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] The present invention provides a double-branch contour enhancement industrial product defect detection method based on a small sample, such as Figure 1 As shown, the following steps are included:

[0037] S1: Collect various types of defect-free and defective industrial product images to build positive sample image datasets and defect sample image datasets of industrial products.

[0038] S2: Construct a background removal module, input two different positive sample images of the same industrial product, one as the reference image and the other as the image to be registered, use the Otsu binarization method and the Canny edge detection operator to remove the background of the reference image and the image to be registered respectively, and obtain the first reference image and the first image to be registered;

[0039] The Otsu binarization method and the Canny edge detection operator are used to remove the background of the reference image and the image to be registered, respectively. Specifically:

[0040] S2.1: Extract the target objects in the reference image and the image to be registered using the Otsu binarization method;

[0041] S2.2: Use the Canny edge detection operator to extract the edge information of the target object, and use the edge information to constrain the target object: starting from the surrounding edge pixels of the reference image and the image to be registered, traverse each pixel inward, and before encountering the edge pixel, set the traversed pixels as the background. After the traversal is completed, the background removal of the reference image and the image to be registered is achieved.

[0042] S3: constructing an image rotation coarse registration module, rotating the first image to be registered to be aligned with the first reference image, and obtaining a second image to be registered;

[0043] The first image to be registered is rotated to be aligned with the first reference image, specifically: the central axis angle α of the circumscribed rectangle of the maximum connected area of ​​the first reference image is calculated, the central axis angle β of the circumscribed rectangle of the maximum connected area of ​​the first image to be registered is calculated, and by comparing the angle difference between the central axis angle α and the central axis angle β, a rotation matrix is ​​constructed to rotate the first image to be registered to be aligned with the first reference image, thereby achieving a coarse alignment of the first image to be registered and the first reference image at the image level.

[0044] S4: Construct a dual-branch feature-level precise registration module, and after training with a positive sample image dataset, input the first reference image and the second image to be registered for precise alignment, obtain the feature map a of the first reference image and the feature map b of the second image to be registered, upsample the feature map a and the feature map b and perform channel dimension splicing to obtain the feature map c, use the multivariate Gaussian distribution estimator to solve the mean and variance of the feature map c as the template feature of the industrial product;

[0045] The dual-branch feature-level precise registration module is specifically constructed by building a dual-subnet structure based on the ResNet-18 network. The two subnets share parameters. Each subnet adds a dual-branch contour enhancement spatial transformer network submodule after the first convolution block, and adds a spatial transformer network STN submodule after the second and third convolution blocks. The loss function uses a negative cosine loss function.

[0046] The dual-branch contour enhancement spatial transformation network submodule consists of two branches: the first branch is the main branch, and its input is the original image fused with contour information. It is composed of two convolutions with a kernel size of 3×3, and each convolution layer is followed by a maximum pooling layer; the second branch is the auxiliary branch, and its input is an image containing only contour information extracted from the original image using the Canny edge detection operator. It is composed of convolutions with a kernel size of 3×3, and each convolution layer is followed by a maximum pooling layer; the main branch and the auxiliary branch process these two inputs respectively, extract spatial features through convolutional neural networks, and then fuse these spatial features through a fully connected layer to predict the transformation parameters and construct a transformation matrix to align the image.

[0047] S5: Solve the feature map d of the defect sample image of the same industrial product, calculate the Mahalanobis distance between the feature map d and the template feature, and use it as the abnormal score matrix of the defect sample image;

[0048] The feature map d of the defective sample image of the same industrial product is solved as follows: the reference image and a defective sample image of the same industrial product are processed by the background removal module, the image rotation coarse registration module and the dual-branch feature-level fine registration module respectively, and then up-sampled and spliced ​​in the channel dimension to obtain the feature map d of the defective sample image.

[0049] S6: Establish an adaptive defect threshold segmentation network, solve the optimal global segmentation threshold, use the global segmentation threshold to segment the anomaly score matrix, and obtain a defect segmentation image;

[0050] The optimal global segmentation threshold is solved by inputting the anomaly score matrix and the corresponding defect true mask map into the adaptive defect threshold segmentation network, using the neural network to automatically determine the defect threshold, subtracting the defect threshold from the anomaly score matrix, and using the activation function ReLU and normalization for soft binarization processing to minimize the MSE loss between the predicted soft binary defect segmentation map and the defect true mask map, and obtain the optimal global segmentation threshold for the anomaly score matrix.

[0051] The small sample-based dual-branch contour enhancement industrial product defect detection method provided by the present invention enhances the network's attention to the relative relationship between samples, reduces the dependence on absolute label information and the demand for the number of labeled samples, meets the rapid iteration and diversification needs of industrial production, can quickly adapt to new product characteristics, and efficiently identify various unknown defects that may occur in the production process, ensuring a dual improvement in production efficiency and product quality, and achieving a better balance between detection accuracy and model versatility in small sample scenarios.

[0052] Example 1

[0053] The method provided by the present invention is used to detect defects in tablets, comprising the following steps:

[0054] S1: Construct a dataset of positive sample images and defective sample images of tablets;

[0055] S2: Construct a background removal module, input two different positive sample images of the pill, and one as a reference image (such as Figure 3 As shown in FIG. 1 , one image is used as the image to be registered, and the background of the reference image and the image to be registered are removed by using the Otsu binarization method and the Canny edge detection operator to obtain a first reference image and a first image to be registered;

[0056] S3: constructing an image rotation coarse registration module, rotating the first image to be registered to be aligned with the first reference image, and obtaining a second image to be registered;

[0057] S4: construct a dual-branch feature-level precise registration module, and after training with a positive sample image dataset, input the first reference image and the second image to be registered for precise alignment, obtain the feature map a of the first reference image and the feature map b of the second image to be registered, upsample the feature map a and the feature map b and perform channel dimension splicing to obtain the feature map c, solve the mean and variance of the feature map c, and use it as the template feature of the tablet;

[0058] S5: The reference image of the pill (such as Figure 3 ) and a defect sample image (as shown Figure 2 As shown in the figure, the image is processed by the background removal module, the image rotation coarse registration module and the dual-branch feature-level fine registration module respectively, and then the feature map d of the defect sample image is obtained after upsampling and channel dimension splicing. The Mahalanobis distance between the feature map d and the template feature is calculated as the abnormal score matrix of the defect sample image;

[0059] S6: Establish an adaptive defect threshold segmentation network, solve the optimal global segmentation threshold, use the global segmentation threshold to segment the anomaly score matrix, and obtain the defect segmentation image of the tablet, such as Figure 4 shown.

[0060] Example 2

[0061] The method provided by the present invention is used to detect defects in cable products, comprising the following steps:

[0062] S1: Construct positive sample image dataset and defect sample image dataset of cable products;

[0063] S2: Construct a background removal module, input two different positive sample images of cable products, one as a reference image (such as Figure 6 As shown in FIG. 1 , one image is used as the image to be registered, and the background of the reference image and the image to be registered are removed by using the Otsu binarization method and the Canny edge detection operator to obtain a first reference image and a first image to be registered;

[0064] S3: constructing an image rotation coarse registration module, rotating the first image to be registered to be aligned with the first reference image, and obtaining a second image to be registered;

[0065] S4: Construct a dual-branch feature-level precise registration module, and after training with a positive sample image dataset, input the first reference image and the second image to be registered for precise alignment, obtain the feature map a of the first reference image and the feature map b of the second image to be registered, upsample the feature map a and the feature map b and perform channel dimension splicing to obtain the feature map c, solve the mean and variance of the feature map c, and use it as the template feature of the cable product;

[0066] S5: The reference image of the cable product (such as Figure 6 ) and a defect sample image (as shown Figure 5 As shown in the figure, the image is processed by the background removal module, the image rotation coarse registration module and the dual-branch feature-level fine registration module respectively, and then the feature map d of the defect sample image is obtained after upsampling and channel dimension splicing. The Mahalanobis distance between the feature map d and the template feature is calculated as the abnormal score matrix of the defect sample image;

[0067] S6: Establish an adaptive defect threshold segmentation network, solve the optimal global segmentation threshold, use the global segmentation threshold to segment the abnormal score matrix, and obtain the defect segmentation image of the cable product, such as Figure 7 shown.

[0068] Example 3

[0069] The method provided by the present invention is used to detect defects in metal nuts, comprising the following steps:

[0070] S1: Construct a positive sample image dataset and a defect sample image dataset of metal nuts;

[0071] S2: Construct a background removal module, input two different positive sample images of metal nuts, one as a reference image (such as Fig. 9 As shown in FIG. 1 , one image is used as the image to be registered, and the background of the reference image and the image to be registered are removed by using the Otsu binarization method and the Canny edge detection operator to obtain a first reference image and a first image to be registered;

[0072] S3: constructing an image rotation coarse registration module, rotating the first image to be registered to be aligned with the first reference image, and obtaining a second image to be registered;

[0073] S4: Construct a dual-branch feature-level precise registration module, and after training with a positive sample image dataset, input the first reference image and the second image to be registered for precise alignment, obtain the feature map a of the first reference image and the feature map b of the second image to be registered, upsample the feature map a and the feature map b and perform channel dimension splicing to obtain the feature map c, solve the mean and variance of the feature map c as the template feature of the metal nut;

[0074] S5: The reference image of the metal nut (such as Fig. 9 ) and a defect sample image (as shown Figure 8 As shown in the figure, the image is processed by the background removal module, the image rotation coarse registration module and the dual-branch feature-level fine registration module respectively, and then the feature map d of the defect sample image is obtained after upsampling and channel dimension splicing. The Mahalanobis distance between the feature map d and the template feature is calculated as the abnormal score matrix of the defect sample image;

[0075] S6: Establish an adaptive defect threshold segmentation network, solve the optimal global segmentation threshold, use the global segmentation threshold to segment the anomaly score matrix, and obtain the defect segmentation image of the metal nut, such as Fig.10 shown.

[0076] It can be seen from Examples 1-3 that the defective areas of tablets, cable products, and metal nuts can be accurately detected by the method of the present invention, and the method of the present invention can be adapted to defect detection of different types of industrial products and has a wide range of adaptability to product defect detection.

Claims

1. A dual-branch contour enhancement industrial product defect detection method based on small samples, characterized in that: The following steps are involved: S1: Construct positive sample image dataset and defect sample image dataset of industrial products; S2: Construct a background removal module, input two different positive sample images of the same industrial product, one as the reference image and the other as the image to be registered, use the Otsu binarization method and the Canny edge detection operator to remove the background of the reference image and the image to be registered respectively, and obtain the first reference image and the first image to be registered; S3: constructing an image rotation coarse registration module, rotating the first image to be registered to be aligned with the first reference image, and obtaining a second image to be registered; S4: Construct a dual-branch feature-level precise registration module, and after training with a positive sample image dataset, input the first reference image and the second image to be registered for precise alignment, obtain the feature map a of the first reference image and the feature map b of the second image to be registered, upsample the feature map a and the feature map b and perform channel dimension splicing to obtain the feature map c, solve the mean and variance of the feature map c, and use it as the template feature of the industrial product; S5: Solve the feature map d of the defect sample image of the same industrial product, calculate the Mahalanobis distance between the feature map d and the template feature, and use it as the abnormal score matrix of the defect sample image; S6: Establish an adaptive defect threshold segmentation network, solve the optimal global segmentation threshold, use the global segmentation threshold to segment the anomaly score matrix, and obtain a defect segmentation image.

2. The small sample-based dual-branch contour enhancement industrial product defect detection method according to claim 1 is characterized in that: In S2, the Otsu binarization method and the Canny edge detection operator are used to remove the background of the reference image and the image to be registered, respectively, as follows: S2.1: Extract the target objects in the reference image and the image to be registered using the Otsu binarization method; S2.2: Use the Canny edge detection operator to extract the edge information of the target object, and use the edge information to constrain the target object: starting from the surrounding edge pixels of the reference image and the image to be registered, traverse each pixel inward, and before encountering an edge pixel, set the traversed pixels to the background.

3. The small sample-based dual-branch contour enhancement industrial product defect detection method according to claim 1, characterized in that: In S3, the first image to be registered is rotated to be aligned with the first reference image, specifically: the central axis angle α of the circumscribed rectangle of the maximum connected area of ​​the first reference image is calculated, the central axis angle β of the circumscribed rectangle of the maximum connected area of ​​the first image to be registered is calculated, and by comparing the angle difference between the central axis angle α and the central axis angle β, a rotation matrix is ​​constructed to rotate the first image to be registered to be aligned with the first reference image.

4. The small sample-based dual-branch contour enhancement industrial product defect detection method according to claim 1, characterized in that: The dual-branch feature-level precise registration module in S4 is specifically, based on the ResNet-18 network, a dual-subnet structure is constructed, the two subnet parameters are shared, each subnet adds a dual-branch contour enhancement spatial transformation network submodule after the first convolution block, and adds a spatial transformation network STN submodule after the second and third convolution blocks, and the loss function adopts a negative cosine loss function; The mean and variance of the characteristic graph c are solved in S4 by using a multivariate Gaussian distribution estimator.

5. The small sample-based dual-branch contour enhancement industrial product defect detection method according to claim 4 is characterized in that: The dual-branch contour enhancement spatial transformation network submodule consists of two branches: the first branch is the main branch, and the input is the original image fused with contour information, which is composed of two convolutions with a kernel size of 3×3, and each convolution layer is followed by a maximum pooling layer; the second branch is the auxiliary branch, and the input is an image containing only contour information extracted from the original image using the Canny edge detection operator, which is composed of convolutions with a kernel size of 3×3, and each convolution layer is followed by a maximum pooling layer; the main branch and the auxiliary branch process the two inputs respectively, extract spatial features through a convolutional neural network, and then fuse these spatial features through a fully connected layer to predict transformation parameters and construct a transformation matrix to perform alignment transformation on the image.

6. The small sample-based dual-branch contour enhancement industrial product defect detection method according to claim 1, characterized in that: The feature map d of the defective sample image of the same industrial product is solved in S5, specifically: the reference image and a defective sample image of the same industrial product are processed by a background removal module, an image rotation coarse registration module and a dual-branch feature-level fine registration module respectively, and then up-sampled and spliced ​​in the channel dimension to obtain the feature map d of the defective sample image.

7. The small sample-based dual-branch contour enhancement industrial product defect detection method according to claim 1, characterized in that: The optimal global segmentation threshold is solved in S6, specifically, the anomaly score matrix and the corresponding defect true mask map are input into the adaptive defect threshold segmentation network, the defect threshold is automatically determined by the neural network, the defect threshold is subtracted from the anomaly score matrix, and soft binarization is performed using the activation function ReLU and normalization to minimize the MSE loss between the predicted soft binary defect segmentation map and the defect true mask map, so as to obtain the optimal global segmentation threshold of the anomaly score matrix.

Images